Smart LLM routing across latency, cost, and reliability
Smart LLM routing means the gateway, not your application, decides which upstream route serves each model call. On Router One a request that names an exact model follows the default model_name strategy and stays on that model; a request with model="auto" is served from a server-owned candidate set scored on recent EWMA latency (alpha 0.20, 15-minute metric TTL, keyed per model/provider pair), posted per-model cost, and recent reliability. One OpenAI-compatible endpoint fronts 40+ models, and the request trace records the model and route that answered. The alternative — hardcoding one provider — means inheriting its bad days: latency spikes, rate limits, and regional incidents.
The signals behind every decision
EWMA latency
An exponentially weighted moving average keyed by model and provider, with alpha=0.20. Metric entries expire after 15 minutes so stale observations do not remain authoritative; this is not a fixed last-50-request window.
Reliability
Observed success and failure outcomes provide a reliability signal when a supported candidate-routing path is active. There is no public promise of a customer-set threshold.
Posted cost
The token-level cost of each available candidate can be considered alongside latency and reliability rather than treated as a universal lowest-price rule.
Routing mode boundary
Production defaults to model_name. Adaptive candidate evaluation only applies where the selected routing path supports a candidate set; it is not an automatic scoring pass on every request.
Exact model selection
Specify an exact model id in the standard OpenAI-compatible request when model choice must be predictable. Router One does not require an ad hoc router object in the request body.
Same-model fallback
For an exact-model request, an eligible failure can retry another healthy provider route serving that model. The model="auto" path retries within its server-owned candidates. Total latency has no universal 200ms guarantee.
Predictable by design
Routing only helps if you can trust its boundary. Exact-model requests use the default model_name strategy; supported candidate routing uses latency, cost, and reliability rather than a quality score; and eligible provider fallback keeps the requested model unchanged. The current public contract does not promise per-project weights or a project-level fallback switch.
Use model="auto" with the standard request shape
Set model="auto" in the normal OpenAI-compatible body. Candidate selection is server-managed; do not send a custom router object or latency/cost/quality weights.
# POST https://api.router.one/v1/chat/completions { "model": "auto", "messages": [{"role": "user", "content": "Hello"}] }
FAQ
How does smart routing decide which route to use?
Exact-model requests use the default model_name strategy. With model="auto", Router One can consider model-provider EWMA latency, posted cost, and reliability/success rate when selecting and retrying its server-owned candidates.
Can I keep using one exact model?
Yes. Specify the model id in the standard OpenAI-compatible request. The default model_name strategy keeps model selection explicit.
What happens when a provider degrades?
Reliability observations can influence model="auto" routing. An eligible exact-model failure can retry another provider route for the same model, while auto routing retries within its server-owned candidates. No fixed fallback time is promised.
Will provider fallback change the requested model?
For exact-model requests, provider fallback keeps the model unchanged. With model="auto", model selection comes from the server-owned candidate set. The trace records the final model and provider route.
Can I configure weights per project or API key?
The current public contract does not expose per-project or per-key latency, cost, or quality weights. Supported candidate routing uses gateway-managed signals.
Where do I see the final routing result?
Every request appears in the dashboard with its final model and provider route, tokens, cost, latency, and status. The methodology behind the signals is documented on the routing methodology page.
Related
Stop hardcoding one provider's bad days.
Get your API key