Connect the Haystack OpenAIChatGenerator to Router One with one base URL

Install haystack-ai and set your credentials

Use Python 3.10 or newer. Install haystack-ai: OpenAIChatGenerator and the OpenAI Python SDK it calls are part of the core package, so this guide needs no integration package. The shell example below is for macOS/Linux. Replace both placeholders before running it, using a Router One key and the exact ID of a current model that supports Chat Completions. The ROUTER_ONE_* names belong to this example, which reads them explicitly; OPENAI_API_KEY is not needed, because the generator receives its own Secret. Keep the same environment active when running the Python file.

python -m pip install haystack-ai
export ROUTER_ONE_API_KEY="sk-your-router-one-key"
export ROUTER_ONE_MODEL_ID="<exact-model-id-from-/models>"

Configure Haystack to use the Router One base URL

Save this as haystack_router_one.py and run python haystack_router_one.py. Secret.from_env_var("ROUTER_ONE_API_KEY") replaces the default OPENAI_API_KEY Secret of the generator; it is resolved when the generator is constructed, and a missing variable raises a ValueError that names it. model takes the catalog ID unchanged, including any provider prefix that is part of the ID. api_base_url is the /v1 base URL; the OpenAI client inside the generator appends /chat/completions, so each run() call is one POST /v1/chat/completions. The first run() sends one user ChatMessage and reads the reply from replies[0].text; since Haystack 3.0 a plain string is accepted as well. The second run() reuses the same generator with streaming_callback=print_streaming_chunk, which prints every chunk as it arrives; the callback can also be set when the generator is constructed. Nothing else differs between the two calls, so any difference in behavior isolates streaming.

import os

from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.components.generators.utils import print_streaming_chunk
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret

llm = OpenAIChatGenerator(
    api_key=Secret.from_env_var("ROUTER_ONE_API_KEY"),
    model=os.environ["ROUTER_ONE_MODEL_ID"],
    api_base_url="https://api.router.one/v1",
)

# 1. One plain Chat Completions request; the reply is a ChatMessage
result = llm.run(messages=[ChatMessage.from_user("Reply with one short greeting.")])
print(result["replies"][0].text)

# 2. The same generator, streaming: chunks print as they arrive
llm.run(
    messages=[ChatMessage.from_user("Count from 1 to 5, one number per line.")],
    streaming_callback=print_streaming_chunk,
)

Which Haystack call sends which request

Haystack components do not share one client: every generator instance carries its own api_key, model, and api_base_url, and wrappers such as Agent and LLM take a generator instance rather than a URL. The rows below describe the request that reaches the gateway for each feature of the generator in this guide, and the capability to confirm on the model detail page before relying on it.

What the gateway does not serve, and how to budget a pipeline run

A Haystack RAG pipeline chains an embedder, a document store, a retriever, a ChatPromptBuilder, and a generator; only the generator request reaches Router One. OpenAITextEmbedder and OpenAIDocumentEmbedder call the embeddings API, which the gateway does not serve, so pointing their api_base_url at Router One cannot work. Embed with a local sentence-transformers embedder (pip install sentence-transformers-haystack; these components left the core package in Haystack 3.0) or with another embedding provider and its own key, and keep the same embedding model for indexing and for queries. Cross-encoder rankers and document stores likewise run in your process or against their own services. One pipeline run can also produce several model requests: the OpenAI client inside the generator retries connection errors and 408, 409, 429, and 5xx responses up to max_retries (Haystack default 5, or OPENAI_MAX_RETRIES), every Agent step calls the generator again, and FallbackChatGenerator moves to the next generator on any exception. Each attempt that reaches the gateway is one request in Dashboard → Logs with its own request_id, trace, and charge. Give the pipeline a dedicated key with maxSpend, lower max_retries for batch jobs, cap max_agent_steps, and reconcile the token_usage and step_count reported by the Agent against Logs. The default 30-second timeout (OPENAI_TIMEOUT) can be raised with timeout= for slow models. Router One records model-call metadata; it does not run your tools, store documents, or hold pipeline state.

Which model ID should Haystack send?

Copy the exact model ID from /models, preserving case, hyphens, and version suffixes; do not substitute a display name. Open its detail page and match the supported API endpoints, context window, and capabilities such as tool calling to the provider and features selected in Haystack. A catalog listing does not mean the client can use every feature of that model. Give each tool a dedicated API key with a maxSpend cap.

Which API protocol is Haystack using?

OpenAI-compatible describes an interface format; it does not make Chat Completions (/v1/chat/completions), Responses (/v1/responses), and Anthropic Messages (/v1/messages) interchangeable. Check the installed client version, provider configuration, and actual request path against the model detail page and API compatibility fact sheet. A successful plain-text chat does not establish support for hosted tools, conversation state, or file-editing features.

Verify the Haystack call in your request trace

Send a simple text request from Haystack, then match its trace in Dashboard → Logs by time, model, and request_id: tokens, cost, latency, and status. Next, test streaming, tool calls, and multi-turn history separately. For failures, retain the actual request path, full error message, and request_id. If there is no matching log, check client configuration and connectivity before attributing the error to the gateway or upstream.

FAQ