Verified against Claude · 2026-07-28
Route requests to cheaper models instead of paying frontier prices for every call
A prompt for designing a model-routing policy that sends easy requests to a small model and escalates to a frontier model only on a measurable signal — task type, input length, or the cheap model's own self-reported confidence — with a required quality check on the un-escalated traffic to catch silent failures a low escalation rate would hide.
The prompt
Ready to copy — highlighted parts are example details you can swap.
You are designing a model-routing policy that sends each incoming request to the cheapest model tier that can actually handle it correctly, escalating to a more expensive model only when there's a concrete signal the cheaper tier will fail — not a routing policy based on a vague sense of which requests "seem complex." TRAFFIC PROFILE ~50,000 requests/day to a customer-facing chat assistant: roughly 70% simple FAQ-style questions, 20% multi-step troubleshooting, 10% requests involving code or configuration snippets MODEL TIERS AVAILABLE cheap tier: Claude Haiku-class model; expensive tier: Claude Sonnet 4.6; rare top tier: Claude Opus 4.6 for anything Sonnet itself flags as low-confidence ROUTING SIGNALS Define the actual signal or signals the router checks, not a subjective judgment call: - Task-type classification: if requests can be reliably bucketed by type (faq_lookup, troubleshooting_multistep, code_or_config), and historical data shows one bucket is reliably handled by the cheap tier while another reliably isn't, route by bucket — this is the cheapest and most predictable signal when it applies. - Input complexity proxies: length, number of distinct sub-questions, presence of a request for multi-step reasoning or code — state the specific proxy and the threshold, not "if it looks complicated." - Confidence or uncertainty from the cheap tier itself: run the cheap model first, and escalate based on a measurable signal from its own output, such as a low logprob score if available or a structured self-check the cheap model is asked to perform, rather than assuming the cheap model always "knows" when it's wrong, which it frequently doesn't without being explicitly asked to check. ESCALATION RULES State the exact rule connecting a signal to a routing decision — for example, if task type matches a hard category, OR input length exceeds 600 input tokens tokens, OR the cheap model's self-reported confidence is below 3 out of 5 on the cheap model's own self-rated confidence, escalate. A routing rule that can't be stated this concretely isn't implementable and will end up as an engineer's ad hoc judgment call buried in application code with no visibility. FALLBACK AND OVERRIDE - What happens if the cheap tier is escalated to but the expensive tier also produces a low-confidence or failed result — is there a further escalation path, or does it stop and return the best available answer with a flag that it's uncertain? - State whether a caller can force a specific tier explicitly, bypassing routing, and how that request is distinguished from the default routed path. MEASUREMENT Weekly review: % of traffic on each tier, escalation rate, and a 50-response human-graded sample of un-escalated cheap-tier answers checked against the same rubric used for support QA — define how you'll know the routing policy is actually working: the percentage of requests staying on the cheap tier, the escalation rate, and critically, a sampled quality check on cheap-tier responses that weren't escalated, since a routing policy that looks efficient because escalation is rare is not obviously the same thing as a policy that's actually choosing correctly. OUTPUT FORMAT 1. The routing decision tree, with every threshold as a specific number. 2. The fallback/override rules. 3. The measurement plan with the specific metrics and how often they're reviewed. 4. The estimated cost saving versus routing everything to the top tier, calculated against the traffic profile.
Customize
Optional — swap in your own details for the highlighted parts above.
Why this works
Task-type bucketing works as a routing signal specifically because it's grounded in actual historical accuracy per bucket rather than a subjective sense of which requests look hard. Human intuition about 'complexity' doesn't reliably predict where a smaller model actually fails — a short question can require a genuinely fine-grained factual distinction a cheap model gets wrong, while a long, verbose question can be trivially easy once parsed. Routing by measured historical performance per task type, instead of surface complexity, is what makes the routing rule actually correlate with the thing that matters, which is whether the cheap tier gets that type of request right often enough to be worth trying first. Using the cheap model's own self-reported confidence as an escalation trigger, instead of assuming a small model reliably 'knows' when it's out of its depth, matters because model calibration is a known weak point, and it's specifically weak in the direction of overconfidence on requests the model is about to get wrong, not underconfidence on requests it would have gotten right. A cheap model asked to just answer, with no explicit self-check, will frequently produce a fluent, wrong answer with the same tone of certainty as a correct one; asking it to explicitly rate its own confidence as a separate step, even a crude scale, forces a moment of self-assessment that measurably improves the correlation between stated confidence and actual correctness compared to inferring confidence from fluency alone. The requirement to sample and quality-check un-escalated cheap-tier responses, not just track the escalation rate, exists because escalation rate alone is a classic survivorship-bias trap: a routing policy with a low escalation rate looks efficient whether the cheap tier is actually handling those requests well or silently getting a meaningful fraction of them wrong without ever triggering the low-confidence signal that would have caused an escalation. The only way to distinguish 'the routing is working' from 'the routing is quietly failing in a way nothing measures' is to periodically grade a sample of exactly the responses that never got flagged, which is the population where an undetected failure would actually be hiding.
Verified against
Claude Haiku 4.5 + Sonnet 4.6 tiering · 2026-07-28
LiteLLM 1.58 · 2026-07-28
Changelog
- 2026-07-28 — Initial publish, verified against a Claude Haiku/Sonnet tiered routing setup via LiteLLM 1.58.
Building this for real?
This is a free starting point. If you'd rather have AI agents & automation built and running for your business, that's Scult's day job.
EXPLORE AI AGENTS & AUTOMATION
