Streaming LLM responses over one endpoint

Turn it on

Set stream to true. The -N flag keeps curl from buffering, so chunks print as they arrive. Pick a chat model whose /models entry supports streaming and lists POST /v1/chat/completions:

curl -N https://api.router.one/v1/chat/completions \
  -H "Authorization: Bearer sk-your-router-one-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-id-from-/models>",
    "messages": [{"role": "user", "content": "Write a haiku"}],
    "stream": true
  }'

What comes over the wire

The response is a text/event-stream: each SSE line carries a chat.completion.chunk object whose choices[].delta holds a fragment — the first chunk typically carries the role, later ones carry content pieces, and the stream ends with a literal [DONE]:

data: {"object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant"},"index":0}]}

data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":"Autumn"},"index":0}]}

data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":" moon"},"index":0}]}

data: {"object":"chat.completion.chunk","choices":[{"delta":{},"finish_reason":"stop","index":0}]}

data: [DONE]

Client patterns

The official Python SDK parses SSE, but your app still needs to track how generation ended. Set ROUTER_ONE_API_KEY in your environment. This text-only, single-choice example requests usage, keeps it even when choices is empty, and records the response header's request_id. SDK errors propagate; the finally block preserves diagnostics and the text already printed. Automatic SDK retries are disabled here so another request is an explicit decision:

from openai import OpenAI
import os
import sys

client = OpenAI(
    base_url="https://api.router.one/v1",
    api_key=os.environ["ROUTER_ONE_API_KEY"],
    max_retries=0,
)

request_id = finish_reason = usage = None
try:
    with client.chat.completions.create(
        model="<model-id-from-/models>",
        messages=[{"role": "user", "content": "Write a haiku"}],
        stream=True,
        stream_options={"include_usage": True},
    ) as stream:
        request_id = stream.response.headers.get("x-request-id")
        for chunk in stream:
            if chunk.usage is not None:
                usage = chunk.usage.model_dump()
            if not chunk.choices:
                continue
            choice = chunk.choices[0]
            if choice.delta.content:
                print(choice.delta.content, end="", flush=True)
            if choice.finish_reason is not None:
                finish_reason = choice.finish_reason
finally:
    print(f"\nrequest_id={request_id} finish_reason={finish_reason}",
          file=sys.stderr)
    print(f"usage={usage}", file=sys.stderr)

if finish_reason is None:
    raise RuntimeError("No finish_reason received; keep partial output.")
if finish_reason != "stop":
    print("Inspect the finish reason before using this as a final answer.",
          file=sys.stderr)

• Keep reading after finish_reason: a final usage-only chunk can arrive afterward. If the stream is interrupted, that chunk may never arrive. Missing usage is unknown, not proof of a zero bill; check Dashboard → Logs.
• The SDK consumes [DONE] internally. This example requires a finish_reason before accepting the text generation as ended; ending a Python iterator alone does not establish that. It does not reconstruct tool calls or validate that the answer meets your application's requirements.

Timeouts, cancellation, and long generations

• Separate connection setup, waiting for response headers, waiting for the first content token, gaps between chunks, and total duration. A reasoning model can legitimately pause; silence alone cannot tell you whether it is thinking, queued, or interrupted.
• Choose client timeouts for the model and your application's latency budget. Check the SDK, reverse proxy and hosting platform separately: any of them can close a stream, even when the gateway permits a longer wait. Avoid a short read/idle timeout that cancels healthy reasoning; preserve the cancellation reason when your own deadline is reached.
• Cancelling is just closing the HTTP connection — no special API. Wire your UI's stop button to abort the request. What you pay for a cancelled request: on POST /v1/chat/completions and POST /v1/responses the gateway keeps reading the upstream for up to five seconds to collect the final usage, records the request as HTTP 499 client_cancelled, bills only the usage the upstream actually reported (if none arrives in that window, only usage already observed — often none), releases the balance it had reserved, and never fails a cancelled request over to another route.
• Native Responses streams for a named model are not cut off by the gateway for waiting: there is no gateway-side response-header or idle timeout on that path, so a long wait before the first event or between events ends only when your client disconnects, the request deadline passes, or the upstream errors or closes the stream. Requests on model:auto keep their per-candidate time budget.
• Compare end-to-end duration in Dashboard → Logs to identify slow requests. Separating network time from model generation requires client-side time-to-first-token and network measurements; the total alone cannot distinguish them.

How do I know an SSE response actually completed?

HTTP 200 only confirms that the stream opened. Read the protocol's terminal signal and keep handling errors until the stream closes. Chat Completions and Responses use different event formats; do not feed Responses events into a Chat Completions delta parser. If the connection ends unexpectedly, retain partial output and the request_id before deciding whether to retry.

Check streamed usage and settlement in request logs

Dashboard → Logs shows the model, tokens, settled cost, latency and status for recorded streamed requests, and per-key budgets still apply. Use the recorded usage and settled cost, including any discount shown in the details, to check billing. Requests awaiting pricing may not appear yet. The customer view does not show the provider or intermediate attempts, and status alone does not reveal how many tokens reached your client; retain the stream error and request_id for that investigation.

FAQ