Smart routing methodology
Smart routing is a core Router One capability. This page separates the default exact-model path from supported candidate routing, and documents the signals, fallback behavior, and public configuration boundary.
Last updated:
Routing signals
- Latency (EWMA)
- Exponentially weighted moving average keyed by model and provider, using alpha=0.20. A metric entry expires after 15 minutes, so stale observations do not remain authoritative. This is not a fixed last-50-request window.
- Reliability
- Observed success and failure outcomes contribute a reliability signal for candidate routing. We do not publish a fixed customer-configurable threshold or promise that every request is adaptively scored.
- Cost
- Posted token-level cost can be considered alongside latency and reliability when a supported candidate-routing path is active.
- Routing mode
- Production defaults to the model_name strategy. An exact-model request follows that selection; model="auto" uses a server-owned candidate set and does not accept customer-defined weights or a router object.
Fallback behavior
- Trigger
- When an eligible upstream attempt fails and another candidate is available, the gateway can retry the request under its routing policy.
- Same requested model
- For an exact-model request, fallback can try another healthy provider route serving that same model and does not substitute another model variant. The model="auto" path instead retries within its server-owned candidate set.
- Fallback latency
- End-to-end latency includes the failed attempt and any subsequent candidate attempt. It varies with the upstream failure mode and response time; Router One does not publish a universal 200ms fallback guarantee.
- Bounded retries
- Retries are limited by the available candidate set and gateway policy. The public product contract does not expose a per-project retry cap.
What appears in your trace
- Provider used
- The final upstream provider and model route that produced the request result.
- Final route
- The trace identifies the model and provider route that produced the request result. It does not promise a customer-visible chain for every failed intermediate attempt.
- Token counts and cost
- Input, output, and cached-input token counts, plus the computed cost at the rate that applied at request time.
Configuration boundary
- Exact model selection
- Specify the model you want in the standard OpenAI-compatible request. The default model_name strategy does not reinterpret every request as an adaptive optimization problem.
- Adaptive candidate routing
- With model="auto", Router One owns the candidate set, can compare latency, cost, and reliability, and can retry eligible failures within the global auto-route budget.
- No implied custom controls
- This methodology does not promise per-project or per-key weights, quality scores, a request-body router object, provider allowlists, or a project-level fallback switch.
FAQ
Can I configure routing weights per project or API key?
The current public product contract does not expose per-project or per-key latency, cost, or quality weights. Supported candidate routing uses gateway-managed latency, cost, and reliability signals.
Is Router One swapping models silently?
No. Provider fallback stays on the requested model. The trace records the final model and provider route; the public contract does not promise substitution to another model variant.
How quickly do routing decisions adapt to new conditions?
For supported candidate routing, latency uses an EWMA keyed by model and provider with alpha=0.20, and entries expire after 15 minutes. Adaptation time depends on incoming observations; it is not guaranteed in a fixed number of seconds or requests.