Understanding AI Model Routing: Latency vs Cost vs Quality

When your application sends a request to an LLM, someone has to decide which model handles it. In a simple setup, that decision is hardcoded: every request goes to the same model at the same provider. In a production system handling thousands of requests per minute across different use cases, that approach leaves performance and money on the table.

Intelligent model routing is the practice of selecting a model-provider route using a defined routing mode and current operating signals. This article explains both the general idea and the narrower contract Router One exposes today.

What Is Model Routing?

Model routing is the decision layer that sits between your application and the LLM providers. Depending on the routing mode, it can honor an exact model selection or evaluate a supported set of candidates.

The simplest form is a static mapping: "always use Claude Sonnet." Router One production traffic defaults to the model_name strategy, so an exact-model request does not automatically become an adaptive scoring exercise. With model="auto", the gateway uses a server-owned candidate set and can use latency, cost, and reliability signals to choose and retry candidates.

The value of routing increases with the number of models and providers you have access to. With one model, there is nothing to route. With ten models across four providers, the routing decision can meaningfully impact cost, speed, and reliability on every single request.

The Three Candidate-Routing Signals

Router One's supported candidate-routing paths can use three operational signals. These signals do not imply that every request is adaptively scored.

Latency

How fast will this model respond? Latency varies not just between models but between providers of the same model, and it fluctuates throughout the day based on load. A model that responds in 200 milliseconds at 2 AM might take 2 seconds at peak hours.

Observed latency can help distinguish model-provider routes under current load. This matters for real-time applications such as chatbots, autocomplete, and interactive agents where users are waiting.

Cost

How much will this request cost? Pricing varies dramatically across models and providers. A request that costs $0.001 on a small model might cost $0.05 on a frontier model — a 50x difference. For high-volume workloads, this difference translates directly to thousands of dollars per month.

Posted token pricing gives the router a comparable cost signal. It can be considered with latency and reliability, but Router One does not promise that the cheapest route always wins.

Reliability

Did recent attempts on this model-provider route succeed? Observed successes and failures contribute a reliability signal for supported candidate routing and eligible retries.

Response quality remains an application-level model-selection concern. Router One's public routing contract does not expose a quality score or a customer-configurable quality weight.

How EWMA Scoring Works for Latency Tracking

Static latency benchmarks are nearly useless for routing decisions because provider performance changes constantly. What you need is a real-time signal that adapts to current conditions while smoothing out noise from individual request variance.

This is exactly what EWMA — Exponentially Weighted Moving Average — provides.

EWMA calculates a running average of observed latencies where recent measurements carry more weight than older ones. The formula is straightforward:

EWMA_new = alpha * latency_observed + (1 - alpha) * EWMA_previous

Router One uses alpha=0.20. A metric entry is keyed by model and provider and expires after 15 minutes, preventing an old observation from remaining authoritative indefinitely. This is not a fixed last-50-request window.

When a supported candidate-routing path is active, current EWMA observations can help compare available model-provider routes. Exact-model requests still follow the default model_name strategy rather than being automatically re-scored across all models.

This approach gives candidate routing a recent latency signal while allowing stale entries to expire.

Default and Candidate Routing Modes

Router One deliberately separates exact model selection from adaptive candidate routing:

• Default model_name strategy — the model id in the standard OpenAI-compatible request remains the model-selection contract.
• model="auto" routing — the gateway owns the candidate set, can compare latency, posted cost, and reliability, and can retry eligible failures within the global auto-route budget.

The public contract does not expose latency/cost/quality weights per project or API key, a quality score, or a custom router object in the request body. Those controls should not be inferred from the existence of internal routing signals.

Automatic Failover: Detecting Failures and Rerouting

Provider outages and degradations are not edge cases — they are regular events. Any production system that depends on a single provider without failover is accepting unnecessary downtime risk.

Router One's candidate retry path has three relevant boundaries:

Reliability signal. Observed successes and failures can influence supported candidate routing. Router One does not publish a fixed rule such as "three 500 errors in ten seconds" as a customer-configurable contract.

Candidate retry. When an upstream failure is eligible for retry and another candidate is available, the gateway can attempt that candidate automatically within the routing policy.

Signal freshness. Latency EWMA entries expire after 15 minutes and are rebuilt from new observations. The public contract does not promise a particular active-probe or gradual-reintroduction algorithm.

How Router One Implements Intelligent Routing

Router One's routing engine keeps its default behavior explicit: production uses the model_name strategy. An exact-model request therefore preserves model selection instead of being adaptively scored across every available model.

With model="auto", the engine can compare model-provider EWMA latency, posted cost, and reliability and can retry eligible failures across its server-owned candidates. This does not imply a quality score or project/API-key weight controls.

The per-request trace records the final route that served the call, including model, provider, token usage, cost, latency, and status. It does not promise a customer-visible chain for every failed intermediate attempt or a configurable scorecard.

This gives operators evidence of what happened while keeping the public routing contract narrower than the internal implementation.

Start Routing Smarter

If you are sending all LLM traffic to a single provider, supported candidate routing can reduce that dependency while keeping exact-model requests predictable.

Sign up at router.one, keep the standard OpenAI-compatible request shape, and use the documented routing modes and traces to evaluate behavior with your own workload.