Streaming LLM responses over one endpoint

Turn it on

Set stream to true. The -N flag keeps curl from buffering, so chunks print as they arrive. Pick a chat model whose /models entry supports streaming and lists POST /v1/chat/completions:

curl -N https://api.router.one/v1/chat/completions \
  -H "Authorization: Bearer sk-your-router-one-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-id-from-/models>",
    "messages": [{"role": "user", "content": "Write a haiku"}],
    "stream": true
  }'

What comes over the wire

The response is a text/event-stream: each SSE line carries a chat.completion.chunk object whose choices[].delta holds a fragment — the first chunk typically carries the role, later ones carry content pieces, and the stream ends with a literal [DONE]:

data: {"object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant"},"index":0}]}

data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":"Autumn"},"index":0}]}

data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":" moon"},"index":0}]}

data: {"object":"chat.completion.chunk","choices":[{"delta":{},"finish_reason":"stop","index":0}]}

data: [DONE]

Client patterns

The official Python SDK parses SSE, but your app still needs to track how generation ended. Set ROUTER_ONE_API_KEY in your environment. This text-only, single-choice example requests usage, keeps it even when choices is empty, and records the response header's request_id. SDK errors propagate; the finally block preserves diagnostics and the text already printed. Automatic SDK retries are disabled here so another request is an explicit decision:

from openai import OpenAI
import os
import sys

client = OpenAI(
    base_url="https://api.router.one/v1",
    api_key=os.environ["ROUTER_ONE_API_KEY"],
    max_retries=0,
)

request_id = finish_reason = usage = None
try:
    with client.chat.completions.create(
        model="<model-id-from-/models>",
        messages=[{"role": "user", "content": "Write a haiku"}],
        stream=True,
        stream_options={"include_usage": True},
    ) as stream:
        request_id = stream.response.headers.get("x-request-id")
        for chunk in stream:
            if chunk.usage is not None:
                usage = chunk.usage.model_dump()
            if not chunk.choices:
                continue
            choice = chunk.choices[0]
            if choice.delta.content:
                print(choice.delta.content, end="", flush=True)
            if choice.finish_reason is not None:
                finish_reason = choice.finish_reason
finally:
    print(f"\nrequest_id={request_id} finish_reason={finish_reason}",
          file=sys.stderr)
    print(f"usage={usage}", file=sys.stderr)

if finish_reason is None:
    raise RuntimeError("No finish_reason received; keep partial output.")
if finish_reason != "stop":
    print("Inspect the finish reason before using this as a final answer.",
          file=sys.stderr)

• Keep reading after finish_reason: a final usage-only chunk can arrive afterward. If the stream is interrupted, that chunk may never arrive. Missing usage is unknown, not proof of a zero bill; check Dashboard → Logs.
• The SDK consumes [DONE] internally. This example requires a finish_reason before accepting the text generation as ended; ending a Python iterator alone does not establish that. It does not reconstruct tool calls or validate that the answer meets your application's requirements.

Timeouts, cancellation, and long generations

• Separate connection setup, waiting for response headers, waiting for the first content token, gaps between chunks, and total duration. A reasoning model can legitimately pause; silence alone cannot tell you whether it is thinking, queued, or interrupted.
• Choose client timeouts for the model and your application's latency budget. Check the SDK, reverse proxy and hosting platform separately: any of them can close a stream, even when the gateway permits a longer wait. Avoid a short read/idle timeout that cancels healthy reasoning; preserve the cancellation reason when your own deadline is reached.
• Cancelling is just closing the HTTP connection — no special API. Wire your UI's stop button to abort the request. What you pay for a cancelled request: on POST /v1/chat/completions and POST /v1/responses the gateway keeps reading the upstream for up to five seconds to collect the final usage, records the request as HTTP 499 client_cancelled, bills the usage the upstream reported, releases the rest of the balance it had reserved, and never fails a cancelled request over to another route. If no final usage arrives in that window and the stream had already produced output, the part the upstream did not report is estimated and billed (as of 2026-10-13) — output tokens from the text already streamed, input tokens from the request; a stream cancelled before any output is not billed if no usage arrives. Dashboard → Logs shows the settled cost, while its token counts show only what the upstream reported, so an estimated charge can appear next to zero tokens.
• A stream: false call is settled differently (as of 2026-10-09): closing the connection does not stop the model on /v1/chat/completions, /v1/responses or /v1/messages — the gateway keeps waiting within its non-streaming window, and if the model finishes, the request is billed for the usage it reports and recorded as HTTP 499 client_cancelled with those tokens; if the model fails or runs past the window, nothing is billed. An SDK timeout on a long non-streaming call can therefore still be billed, and a retry the SDK sends after that timeout is a separate request, billed on its own — per openai-python's README (3.29.0, checked 2026-10-13), it retries a timed-out request twice by default. Use stream: true for generations that take minutes, or set the client timeout above the expected generation time.
• Native Responses streams for a named model are not cut off by the gateway for waiting: there is no gateway-side response-header or idle timeout on that path, so a long wait before the first event or between events ends only when your client disconnects, the request deadline passes, or the upstream errors or closes the stream. Requests on model:auto keep their per-candidate time budget.
• Dashboard → Logs shows each request's total time and, for a streamed request that produced output, its time to first token (TTFT). Both are timed from when the gateway starts executing the request, so TTFT includes any failed attempt before failover. The gap between the first visible token your client measured and the logged TTFT is your network, your client and the gateway's request intake; a long TTFT itself points at queueing, a failed attempt before failover, or the model's own wait.

How do I know an SSE response actually completed?

HTTP 200 only confirms that the stream opened. Read the protocol's terminal signal and keep handling errors until the stream closes. Chat Completions and Responses use different event formats; do not feed Responses events into a Chat Completions delta parser. If the connection ends unexpectedly, retain partial output and the request_id before deciding whether to retry.

Check streamed usage and settlement in request logs

Dashboard → Logs shows the model, tokens, settled cost, total time — plus time to first token (TTFT) once the stream produced output — and status for recorded streamed requests, and per-key budgets still apply. Use the recorded usage and settled cost, including any discount shown in the details, to check billing. Requests awaiting pricing may not appear yet. The customer view does not show the provider or intermediate attempts, and status alone does not reveal how many tokens reached your client; retain the stream error and request_id for that investigation.

FAQ