Skip to content
Router One
Sign up

Smart LLM routing that keeps the model you name

Smart LLM routing means the gateway, not your application, handles which upstream route serves each model call. On Router One a request that names a model is served by that model: its routes are tried in a gateway-managed order, a route that fails with a 429, a 5xx or a timeout can be retried on another route serving the same model when one is available, and routes with a sustained upstream-error rate are automatically moved to the back of the order until they recover. A request with model="auto" is classified into a low, medium or high tier and served by a model from that tier's server-owned list. One OpenAI-compatible endpoint fronts 30+ models, and Customer Logs records the model, request ID, usage, settled cost, recorded latency, and status. The alternative — hardcoding one provider — means inheriting its bad days: latency spikes, rate limits, and regional incidents.

Get your API key

What happens to each request

The model you name

Production runs the model_name strategy: a request that names a model is served by that model, and only routes serving that same model are candidates. Failover never substitutes another model or variant.

Same-model failover

When a route returns a 429, a 5xx or times out, the gateway can retry the request on another route serving the same model. If no other route completes it, your app receives the error, so keep a bounded backoff.

Cooldowns after failures

A route that returns a 429 cools down (for the upstream retry-after when one is sent) before it takes traffic again, a route that returns a bad-gateway, unavailable or timeout error cools down briefly, and a route that fails repeatedly is taken out for a cooldown, then probed before it returns.

Failing routes move to the back

Since 2026-09-24, a route whose recent upstream errors stay high is ordered last for its model and kept only as a last-resort fallback. Once its error rate recovers, it returns to its normal place in the order.

Tiers for model="auto"

With model="auto", the gateway classifies the request (rules plus a lightweight classifier) into a low, medium or high tier and serves it with a model from that tier's server-owned list. An optional X-Route-Session header keeps a conversation on the same choice for a while.

No routing weights to tune

There is no request-body router object and no per-project or per-key routing weights: route order is managed by the gateway. Total latency has no universal 200ms guarantee.

Predictable by design

Routing only helps if you can trust its boundary. Failover never swaps the model you named for another; failover only moves between routes serving that same model; and only model="auto" chooses the model for you, from a server-owned tier list. The current public contract does not promise per-project weights or a project-level fallback switch.

Use model="auto" with the standard request shape

Set model="auto" in the normal OpenAI-compatible body. The tier and the model are chosen server-side; do not send a custom router object or routing weights. To keep a multi-turn conversation on the same choice, send your own conversation ID in an X-Route-Session header.

request.json
# POST https://api.router.one/v1/chat/completions
{
  "model": "auto",
  "messages": [{"role": "user", "content": "Hello"}]
}

FAQ

How does smart routing decide which route to use?

A request that names a model is served by that model (the default model_name strategy): its routes are tried in a gateway-managed order, and routes with a sustained upstream-error rate are moved to the back until they recover. With model="auto", the gateway first picks a low, medium or high tier for the request, then serves it from that tier's server-owned model list.

Can I keep using one exact model?

Yes. Specify the model id in the standard OpenAI-compatible request. The default model_name strategy keeps model selection explicit.

What happens when a provider degrades?

A route that returns a 429, a 5xx or a timeout can be retried on another route serving the same model. A route that keeps failing is cooled down or moved to the back of the order until it recovers, so later requests try the other routes first. model="auto" requests retry within their tier's candidates under a bounded retry budget. No fixed fallback time is promised.

Will provider fallback change the requested model?

No. For exact-model requests, failover only moves between routes serving the same model. With model="auto", the model comes from the chosen tier's server-owned list. The trace records the model and request ID.

Can I configure weights per project or API key?

No. The current public contract does not expose per-project or per-key routing weights; route order is managed by the gateway.

Where do I check a request result?

Customer Logs shows the recorded model, request ID, tokens, settled cost, recorded latency, status, and any sanitized error. It does not disclose upstream provider identities or service routes. The routing methodology page explains the managed behavior.

Related

Stop hardcoding one provider's bad days.

Get your API key