Streaming LLM responses over one endpoint

Streaming turns a chat completion into Server-Sent Events: tokens render as the model produces them, so users read while the model writes. Router One follows the OpenAI Chat Completions contract — set stream to true and the same code path streams GPT, Claude, Gemini, or Grok models, with support varying by model. This page covers the wire format, the client patterns that hold up in production, and the two operational questions streaming always raises: timeouts and cost accounting.

Get your API key

Turn it on

Set stream to true. The -N flag keeps curl from buffering, so chunks print as they arrive. Pick a chat model whose /models entry supports streaming and lists POST /v1/chat/completions:

stream.sh

curl -N https://api.router.one/v1/chat/completions \
  -H "Authorization: Bearer sk-your-router-one-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-id-from-/models>",
    "messages": [{"role": "user", "content": "Write a haiku"}],
    "stream": true
  }'

What comes over the wire

The response is a text/event-stream: each SSE line carries a chat.completion.chunk object whose choices[].delta holds a fragment — the first chunk typically carries the role, later ones carry content pieces, and the stream ends with a literal [DONE]:

wire-format.txt

data: {"object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant"},"index":0}]}

data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":"Autumn"},"index":0}]}

data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":" moon"},"index":0}]}

data: {"object":"chat.completion.chunk","choices":[{"delta":{},"finish_reason":"stop","index":0}]}

data: [DONE]

Client patterns

The official SDKs hide the SSE parsing — you iterate chunks and concatenate deltas. The same loop works for every model family on the endpoint:

stream.py

from openai import OpenAI

client = OpenAI(
    base_url="https://api.router.one/v1",
    api_key="sk-your-router-one-key",
)

stream = client.chat.completions.create(
    model="<model-id-from-/models>",
    messages=[{"role": "user", "content": "Write a haiku"}],
    stream=True,
)
for chunk in stream:
    if not chunk.choices:
        continue  # Usage-only chunks have no choices.
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Timeouts, cancellation, and long generations

Time-to-first-token and total duration are different budgets. A large prompt or a reasoning-heavy model can think for a while before the first chunk — that silence is normal, not a hang.
Do not set aggressive client timeouts on generation endpoints; they convert healthy long generations into client-side failures. If you must bound something, bound idle time between chunks, generously.
Cancelling is just closing the HTTP connection — no special API. Wire your UI's stop button to abort the request.
Compare end-to-end duration in Dashboard → Logs to identify slow requests. Separating network time from model generation requires client-side time-to-first-token and network measurements; the total alone cannot distinguish them.

How do I know an SSE response actually completed?

HTTP 200 only confirms that the stream opened. Chat Completions clients must handle error events and connection failures alongside deltas; a usage-only chunk can have an empty choices array. Responses clients use a different event format: inspect response.completed, response.failed and response.incomplete, including response.error or incomplete_details. Do not feed Responses events into a Chat Completions delta parser. If the connection ends unexpectedly, retain partial output and the request_id before deciding whether to retry.

Streamed requests are still fully traced

Streamed requests are recorded in Dashboard → Logs with the final model, tokens, cost, latency and status, and per-key budgets still apply. Use the recorded usage and cost to check settlement. Status alone does not reveal how many tokens reached your client or which intermediate routes were attempted; retain the stream error and request_id for that investigation.

FAQ

Does streaming cost more?

No. Token accounting is identical to a non-streamed request for the same content — streaming only changes how the response is delivered. The per-request cost trace shows the same fields either way.

Nothing arrives for several seconds — is the request stuck?

Usually not: the model is processing your prompt before emitting the first token, and long prompts or reasoning-heavy models stretch that phase. Resist adding a short timeout — check the request's end-to-end latency in Dashboard → Logs before concluding anything is wrong.

Can tool calls be streamed?

Yes — tool-call fragments arrive incrementally in the delta chunks: accumulate the id, name, and argument pieces until the stream finishes with finish_reason tool_calls, then run the normal tool loop. See the LLM tool calling guide.

What happens if the upstream fails mid-stream?

Retryable upstream failures can move to another eligible route before generation is underway; once you are receiving tokens, a hard upstream failure surfaces as a terminated stream that your client should handle. The automatic fallback page explains which failures are retryable.

Do all models stream?

Router One follows the OpenAI Chat Completions contract, including streaming, with support varying by underlying model. Check the model's entry in the /models catalog, and the docs for current per-feature details.

Get your API key