Every LLM provider has bad hours. Rate limits tighten during launches, regional incidents take endpoints offline, and latency quietly doubles under load long before a status page admits anything. If your application talks to exactly one provider, all of that becomes your incident too.
Fallback is the standard answer, but naive fallback creates its own problems: responses that change personality mid-conversation, retry storms that amplify outages, and failovers you cannot prove happened. This post covers the patterns that hold up in production — and the constraints that make fallback trustworthy rather than just busy.
What should actually trigger a fallback
Not every error deserves a retry elsewhere. Three signals are worth acting on:
- 5xx responses — the upstream failed; trying the same route again mostly re-queues you into the same incident.
- Network errors and timeouts — a request that exceeds its per-model time budget is gone; waiting longer rarely rescues it.
- Degradation trends — rising error rates and latency over a rolling window justify routing around a provider before individual requests fail.
Notably absent: 4xx errors. A malformed request will be malformed everywhere, and retrying a 401 with a different provider is just spending money on the same bug. Router One's gateway triggers fallback on 5xx, network errors, and per-model timeout budgets, and uses rolling error-rate windows to down-rank degrading routes before they fail outright — the exact signals are documented in the routing methodology.
Stay in the model family
The biggest practical decision is what to fall back to. Swapping a Claude request to a GPT model technically returns a response, but it changes tone, formatting behavior, tool-calling quirks, and prompt-sensitivity in the middle of someone's session — a silent quality incident layered on top of the availability one.
The safer contract is same-family failover: a GPT request fails over to another GPT-family route, a Claude request to another Claude-family route. Capability stays consistent, prompts keep working, and the application above never has to care. This is the contract Router One enforces — fallback never crosses model families, and the exact variant that answered is recorded per request. If no healthy same-family route exists, you get a clean error instead of a surprise model swap.
Budget the handoff
Fallback that doubles your latency is a degradation of its own. Two numbers keep it honest:
- Per-attempt timeout — how long a route gets before the gateway gives up on it.
- Handoff overhead — the added cost of switching; through Router One a typical end-to-end fallback adds under 200ms on top of the failing attempt.
For latency-sensitive endpoints, that overhead budget is the difference between "users noticed nothing" and "the fallback was the outage."
If you can't verify it, it doesn't exist
The most common fallback failure is silent: the config looks right, and nobody can prove it ever fired. Every fallback should leave a record — which route failed, with what error code and latency, and which route completed the request. Through the gateway, that record is the per-request trace: both attempts appear, so you can count fallbacks per day, per model family, and per provider, and verify the mechanism during real incidents instead of trusting it.
Teams with compliance or evaluation constraints sometimes need the opposite guarantee — that a request never silently moves. That should be a supported configuration, not a hack: enterprise contracts can disable fallback per project while keeping the same tracing.
The production checklist
- Fallback triggers on 5xx, network errors, and timeout budgets — not on 4xx.
- Failover stays within the model family; no silent personality swaps.
- Handoff overhead is bounded and known (under 200ms through the gateway).
- Every fallback is visible in a per-request trace you can audit later.
- Projects that must not fail over can turn it off explicitly.
The LLM fallback and smart routing pages describe how the pieces fit together, and the China latency benchmark shows the measurement discipline behind the routing claims. One base-URL change at router.one puts the whole pattern in front of your existing code.