Smart routing methodology
Smart routing is a core Router One capability. This page separates the default exact-model path from supported candidate routing, and documents the signals, fallback behavior, and public configuration boundary.
Last updated:
Routing signals
- Latency (EWMA)
- Exponentially weighted moving average keyed by model and provider, using alpha=0.20. A metric entry expires after 15 minutes, so stale observations do not remain authoritative. This is not a fixed last-50-request window.
- Reliability
- Observed success and failure outcomes contribute a reliability signal for candidate routing. We do not publish a fixed customer-configurable threshold or promise that every request is adaptively scored.
- Cost
- Posted token-level cost can be considered alongside latency and reliability when a supported candidate-routing path is active.
- Routing mode
- Production defaults to the model_name strategy. An exact-model request follows that selection; model="auto" uses a server-owned candidate set and does not accept customer-defined weights or a router object.
Fallback behavior
- Trigger
- When an eligible upstream attempt fails and another candidate is available, the gateway can retry the request under its routing policy.
- Same requested model
- For an exact-model request, fallback can try another healthy provider route serving that same model and does not substitute another model variant. The model="auto" path instead retries within its server-owned candidate set.
- Fallback latency
- End-to-end latency includes the failed attempt and any subsequent candidate attempt. It varies with the upstream failure mode and response time; Router One does not publish a universal 200ms fallback guarantee.
- Bounded retries
- Retries are limited by the available candidate set and gateway policy. The public product contract does not expose a per-project retry cap.
What appears in your trace
- Provider used
- The final upstream provider and model route that produced the request result.
- Final route
- The trace identifies the model and provider route that produced the request result. It does not promise a customer-visible chain for every failed intermediate attempt.
- Token counts and cost
- Input, output, and cached-input token counts, plus the computed cost at the rate that applied at request time.
Configuration boundary
- Exact model selection
- Specify the model you want in the standard OpenAI-compatible request. The default model_name strategy does not reinterpret every request as an adaptive optimization problem.
- Adaptive candidate routing
- With model="auto", Router One owns the candidate set, can compare latency, cost, and reliability, and can retry eligible failures within the global auto-route budget.
- No implied custom controls
- This methodology does not promise per-project or per-key weights, quality scores, a request-body router object, provider allowlists, or a project-level fallback switch.
FAQ
- Can I configure routing weights per project or API key?
- The current public product contract does not expose per-project or per-key latency, cost, or quality weights. Supported candidate routing uses gateway-managed latency, cost, and reliability signals.
- Is Router One swapping models silently?
- No. Provider fallback stays on the requested model. The trace records the final model and provider route; the public contract does not promise substitution to another model variant.
- How quickly do routing decisions adapt to new conditions?
- For supported candidate routing, latency uses an EWMA keyed by model and provider with alpha=0.20, and entries expire after 15 minutes. Adaptation time depends on incoming observations; it is not guaranteed in a fixed number of seconds or requests.