When your application sends a request to an LLM, someone has to decide which model handles it. In a simple setup, that decision is hardcoded: every request goes to the same model at the same provider. In a production system handling thousands of requests per minute across different use cases, that approach leaves performance and money on the table.
Intelligent model routing is the practice of selecting a model-provider route using a defined routing mode and current operating signals. This article explains both the general idea and the narrower contract Router One exposes today.
What Is Model Routing?
Model routing is the decision layer that sits between your application and the LLM providers. Depending on the routing mode, it can honor an exact model selection or evaluate a supported set of candidates.
The simplest form is a static mapping: "always use Claude Sonnet." Router One production traffic defaults to the model_name strategy, so an exact-model request does not automatically become an adaptive scoring exercise. With model="auto", the gateway uses a server-owned candidate set and can use latency, cost, and reliability signals to choose and retry candidates.
The value of routing increases with the number of models and providers you have access to. With one model, there is nothing to route. With ten models across four providers, the routing decision can meaningfully impact cost, speed, and reliability on every single request.
The Three Candidate-Routing Signals
Router One's supported candidate-routing paths can use three operational signals. These signals do not imply that every request is adaptively scored.
Latency
How fast will this model respond? Latency varies not just between models but between providers of the same model, and it fluctuates throughout the day based on load. A model that responds in 200 milliseconds at 2 AM might take 2 seconds at peak hours.
Observed latency can help distinguish model-provider routes under current load. This matters for real-time applications such as chatbots, autocomplete, and interactive agents where users are waiting.
Cost
How much will this request cost? Pricing varies dramatically across models and providers. A request that costs $0.001 on a small model might cost $0.05 on a frontier model — a 50x difference. For high-volume workloads, this difference translates directly to thousands of dollars per month.
Posted token pricing gives the router a comparable cost signal. It can be considered with latency and reliability, but Router One does not promise that the cheapest route always wins.
Reliability
Did recent attempts on this model-provider route succeed? Observed successes and failures contribute a reliability signal for supported candidate routing and eligible retries.
Response quality remains an application-level model-selection concern. Router One's public routing contract does not expose a quality score or a customer-configurable quality weight.
How EWMA Scoring Works for Latency Tracking
Static latency benchmarks are nearly useless for routing decisions because provider performance changes constantly. What you need is a real-time signal that adapts to current conditions while smoothing out noise from individual request variance.
This is exactly what EWMA — Exponentially Weighted Moving Average — provides.
EWMA calculates a running average of observed latencies where recent measurements carry more weight than older ones. The formula is straightforward:
EWMA_new = alpha * latency_observed + (1 - alpha) * EWMA_previous
Router One uses alpha=0.20. A metric entry is keyed by model and provider and expires after 15 minutes, preventing an old observation from remaining authoritative indefinitely. This is not a fixed last-50-request window.
When a supported candidate-routing path is active, current EWMA observations can help compare available model-provider routes. Exact-model requests still follow the default model_name strategy rather than being automatically re-scored across all models.
This approach gives candidate routing a recent latency signal while allowing stale entries to expire.
Default and Candidate Routing Modes
Router One deliberately separates exact model selection from adaptive candidate routing:
- Default
model_namestrategy — the model id in the standard OpenAI-compatible request remains the model-selection contract. model="auto"routing — the gateway owns the candidate set, can compare latency, posted cost, and reliability, and can retry eligible failures within the global auto-route budget.
The public contract does not expose latency/cost/quality weights per project or API key, a quality score, or a custom router object in the request body. Those controls should not be inferred from the existence of internal routing signals.
Automatic Failover: Detecting Failures and Rerouting
Provider outages and degradations are not edge cases — they are regular events. Any production system that depends on a single provider without failover is accepting unnecessary downtime risk.
Router One's candidate retry path has three relevant boundaries:
Reliability signal. Observed successes and failures can influence supported candidate routing. Router One does not publish a fixed rule such as "three 500 errors in ten seconds" as a customer-configurable contract.
Candidate retry. When an upstream failure is eligible for retry and another candidate is available, the gateway can attempt that candidate automatically within the routing policy.
Signal freshness. Latency EWMA entries expire after 15 minutes and are rebuilt from new observations. The public contract does not promise a particular active-probe or gradual-reintroduction algorithm.
How Router One Implements Intelligent Routing
Router One's routing engine keeps its default behavior explicit: production uses the model_name strategy. An exact-model request therefore preserves model selection instead of being adaptively scored across every available model.
With model="auto", the engine can compare model-provider EWMA latency, posted cost, and reliability and can retry eligible failures across its server-owned candidates. This does not imply a quality score or project/API-key weight controls.
The per-request trace records the final route that served the call, including model, provider, token usage, cost, latency, and status. It does not promise a customer-visible chain for every failed intermediate attempt or a configurable scorecard.
This gives operators evidence of what happened while keeping the public routing contract narrower than the internal implementation. The smart routing page documents the candidate-ranking behavior, automatic LLM fallback covers what happens when a call fails, and routing methodology states what is and is not promised.
FAQ
What is AI model routing? Model routing is the decision layer that sits between your application and the LLM providers. Depending on the routing mode, it can honor an exact model selection or evaluate a supported set of candidates using latency, cost, and reliability signals.
Does Router One reroute my request when I ask for an exact model?
No. Production traffic defaults to the model_name strategy, so an exact-model request preserves your model selection instead of being adaptively scored across every available model. Adaptive candidate routing applies when you send model="auto".
Which signals does Router One's candidate routing use? Supported candidate-routing paths can use three operational signals: observed latency, posted cost, and reliability built from observed successes and failures. The public contract does not expose a quality score or customer-configurable routing weights.
How does EWMA latency tracking work in Router One?
EWMA keeps a running average of observed latencies in which recent measurements carry more weight, and Router One uses alpha=0.20. Each metric entry is keyed by model and provider and expires after 15 minutes, so a stale observation does not remain authoritative.
What happens when an upstream call fails? When an upstream failure is eligible for retry and another candidate is available, the gateway can attempt that candidate automatically within the routing policy. Automatic LLM fallback covers what happens when a call fails.
Start Routing Smarter
If you are sending all LLM traffic to a single provider, supported candidate routing can reduce that dependency while keeping exact-model requests predictable.
Sign up at router.one, keep the standard OpenAI-compatible request shape, and use the documented routing modes and traces to evaluate behavior with your own workload.