How to verify your LLM API is serving the real model
Third-party LLM API services vary wildly in quality. Some silently swap the model you paid for, some serve quantized or context-truncated variants, and some disappear with prepaid balances. This guide is a reproducible test checklist that works on any provider — run it on Router One too. We publish per-request traces precisely so you can.
The five failure modes to test for
Most complaints about third-party APIs trace back to one of five patterns. Casual chat rarely exposes them — production workloads always do.
Model swapping
You pay for a frontier model but requests are routed to a cheaper one. This is the hardest to spot casually: small models answer everyday questions convincingly. It shows up in complex reasoning, long-context recall, and tool-calling accuracy.
Silent degradation
The right model family, but quantized, distilled, or run with a silently truncated context window. Long-document tasks get noticeably worse while short prompts look fine.
Fake streaming
The service generates the full response, then replays it as chunks. Time-to-first-token approaches total generation time — a dead giveaway under measurement.
Parameter dropping
temperature, JSON mode, stop sequences, or tool definitions are ignored or stripped in transit. Chat demos survive this; structured-output pipelines do not. Router One forwards response_format and tool definitions as sent — the structured outputs guide shows the exact contract, including what the gateway rejects before a model is called.
Exit scams
Prepaid balances vanish when the operator shuts down overnight. This is a business-model risk, not a technical one — see the red-flag table below.
Weak signals: don't just ask the model who it is
Self-reported identity is the least reliable test. A relay can inject a system prompt that makes any model claim to be any other, and models frequently misstate their own version even when unmodified. Treat the following as weak corroborating signals only:
- Knowledge-cutoff probes: ask about events near the claimed model's training cutoff and compare against the official model's behavior.
- Refusal style and formatting habits: model families have recognizable default tones, but these are promptable.
- Tokenizer quirks: unusual handling of rare Unicode or long repeated strings differs across families — suggestive, never conclusive.
Strong signals: API behavior fingerprints
Behavior under load is much harder to fake than identity. Four tests catch most swapping and degradation:
Context-window probe (needle test)
Embed a marker fact near the start of an input close to the advertised context limit, then ask for it back. A silently truncating service returns a confident answer with the needle missing — it never errors, which is exactly the tell.
Tool-calling fidelity
Send a request with several tools and a strict JSON schema. Swapped or heavily quantized models produce malformed arguments and wrong tool choices at rates the real model does not.
Streaming granularity and TTFT
Real streaming arrives token-by-token with a short time-to-first-token. Buffered fake streaming arrives in large chunks after a long initial wait. Log both across a few runs.
Long-instruction adherence
A system prompt with 8-10 explicit constraints (format, forbidden words, ordering) separates frontier models from cheaper stand-ins quickly and reproducibly.
The A/B test: same prompt, two endpoints
The most conclusive method needs no trust in anyone: run identical requests against the official API and the service under test, then diff the behavior on long-context recall, tool calls, and constraint adherence. A few dollars of credit is enough.
PROMPT='<your long-context needle test>'
# Control: the official endpoint
curl https://api.openai.com/v1/chat/completions \
-H "Authorization: Bearer $OFFICIAL_KEY" \
-d "{\"model\": \"gpt-5.5\", \"messages\": [{\"role\": \"user\", \"content\": \"$PROMPT\"}]}"
# Candidate: the service under test (OpenAI-compatible)
curl https://api.router.one/v1/chat/completions \
-H "Authorization: Bearer $ROUTER_ONE_KEY" \
-d "{\"model\": \"openai/gpt-5.5\", \"messages\": [{\"role\": \"user\", \"content\": \"$PROMPT\"}]}"
# Compare: needle recall, refusal style, tool-call formatting,
# streaming granularity, time-to-first-token.Non-technical red flags: the exit-scam checklist
Technical tests catch swapping; these catch operators who will not be around next quarter. None is proof alone — three or more together is a pattern.
| Red flag | Why it matters | How to check |
|---|---|---|
| Prices below any plausible legitimate cost, sold as unlimited | Metered legitimate supply has a hard cost floor. Sustained pricing far below it usually means shared account pools or reverse-engineered endpoints — which come bundled with swapping, bans, and shutdowns | Run the fingerprint tests above; ask where the subsidy could possibly come from |
| Large prepay bonuses, no refund policy | The business model is deposit float; walking away is cheap | Start with the smallest possible top-up and test withdrawal/refund terms first |
| No status page, no SLA, no legal entity | When it breaks, there is no one to hold accountable | Look for a public status page, terms of service, and a refund policy |
| No per-request usage records | You cannot audit what model you were actually billed for | Demand request-level logs: model, tokens, cost, latency, status |
| Support only in a chat group | Commitments made in group chats are unauditable and disappear with the group | Look for formal documentation and a support channel with history |
Run this checklist on Router One
We built for verifiability instead of asking for trust. Every request through Router One carries a per-request trace — model served, tokens, cost, latency, status — visible in your dashboard logs, so the A/B test above can be cross-checked against what we billed you for. The model you request is the model served.
- Model catalog and posted per-token rates are public at /models — no bundled or opaque pricing.
- Prompt and completion bodies are not retained; only request metadata is logged for billing and routing.
- Public status page and SLA describe fallback behavior and uptime commitments.
- A few dollars of balance is enough to run every test on this page against api.router.one.
FAQ
The model says it's GPT or Claude when I ask. Is that proof?
No. A relay can inject a system prompt that makes any model claim any identity, and even unmodified models regularly misstate their own version. Use behavior fingerprints — context probes, tool-calling fidelity, streaming timing — instead of self-reports.
How can some services sell far below official prices, unlimited?
Metered legitimate supply has a cost floor. Sustained pricing far below that floor, with no usage caps, generally depends on unsustainable sourcing such as shared consumer account pools or reverse-engineered private endpoints. Those setups fail in characteristic ways: model swapping, silent degradation, sudden bans, and exit scams.
What's the fastest single test for context truncation?
The needle test: place a marker fact near the start of an input close to the advertised context limit and ask for it back. A truncating service answers confidently with the needle missing — no error is raised, which is exactly how truncation hides.
How does Router One prove it doesn't swap or degrade models?
Per-request traces show the model served, tokens, cost, latency, and status for every call, cross-checkable against your own A/B tests. Rates are posted per model, and the catalog is public. We designed the audit trail so you don't have to take our word for it.
How much does it cost to run this whole checklist?
A few dollars. Every test on this page works with a minimal balance — long-context probes are the most expensive item, and even those cost well under a dollar per run on most models.