Prompt Caching and Model Routing: Stack the Savings Without Losing Cache

Model routing and prompt caching are not in conflict. You can change which model answers a request and still get cache hits on the input you send it. The two features work on different parts of the request.

Prompt caching reuses the computed state of a stable prefix inside each model. On Anthropic, a cache read costs about a tenth of the base input rate, so a repeated $5-per-million prefix reads back at roughly $0.50. Model routing decides which model computes the request-specific tail of that same prefix. A gateway can pick Sonnet for one turn and Opus for the next while sending both an identical, cached prefix. The cache hit survives the model switch.

Three conditions keep them stacked. Keep the stable prefix byte-identical on every request, so the prefix match holds. Pass the cache controls through the gateway untouched, so the provider knows what to cache. And route within a small set of models, because a cache is per model and each one fills only as you use it. Concentrate traffic and each model stays warm.

This guide explains how prompt caching works, why the stable context must lead the prompt, and how a gateway changes model selection while the cached prefix still hits. It closes with a checklist you can apply to any gateway today.

What prompt caching actually caches

Before routing, it helps to be exact about what caching stores, because the most common objection to it comes from a misunderstanding of that.

When a model processes a prompt, it reads every input token through its attention layers and builds an internal representation of that context. Prompt caching stores that representation for a stable prefix of future prompts. The provider writes the computed state for that prefix into a cache, and a later request that starts with the same prefix skips the recomputation and picks up from where the first one left off. IBM describes it as storing the repeated parts of a prompt, such as instructional content and reference material, so the model does not reprocess those tokens on every call.

Caching is not response caching. It does not save and replay an answer. Two requests that share a cached prefix still produce fresh output, because the model computes a new response from that prefix each time. Caching is also not semantic caching. Semantic caching matches prompts by similarity and can reuse a stored result even when the wording changes. Prompt caching requires an exact prefix match. Change a single character inside the cached region and the cache misses.

Prompt caching is a pure inference optimization on input tokens. It reduces time to first token, it lowers the cost of the repeated prefix, and it depends entirely on the input staying identical from one request to the next.

Why does this matter for routing? Because the cache is keyed to the exact prefix and to the specific model and its tokenizer. The same English sentence tokenizes differently across providers, and often across models from the same provider. A cache entry written by one model cannot be read by a different model. That is where the worry starts.

Why the stable context must lead the prompt

A cache hit depends on a prefix match, so the order of your prompt decides whether you ever get one. Put the content that changes on every request at the front, and the prefix never matches. Put the content that stays the same at the front, and the matching part is exactly what you want to reuse.

Every major provider says the same thing. Place static content such as instructions, tool definitions, schemas, and reference context at the beginning, then place variable, request-specific content at the end. On Anthropic, cache prefixes are created in a clear order: tools, then the system prompt, then messages. On OpenAI, cache hits are only possible for exact prefix matches within a prompt, so the guidance is to keep instructions, tools, and shared context stable and put user-specific content last.

The mechanics back this up. On Anthropic, cache writes happen only at a marked breakpoint, and each write covers everything up to and including that point. A change to any block at or before the breakpoint changes the hash of the prefix, and the next request misses. On OpenAI, a prefix must reach a token minimum before it caches, and a change before the breakpoint invalidates the earlier entry.

The practical rule is a single line. Lead with the system prompt, the tools, the examples, the codebase context, and the RAG reference blocks. Append the user's latest message and any turn-specific content after them. When you structure the prompt this way, the repeated bulk of the input stays eligible to hit the cache, and only the short trailing tail is processed fresh.

If you reverse the order, caching becomes a tax with no benefit. The dynamic message at the front makes the prefix different on every call, so every request re-processes the bulk at the full input rate and still pays the cache-write fee. This is one of the most common silent failures, because nothing reports it. The prompt looks correct and the bill just keeps climbing.

How the discount works and what a hit costs

The discount is the reason anyone cares. On Anthropic, a cache read and a cache refresh run at about a tenth of the base input rate. The pricing table spells it out across the model tiers.

Several details in that table change the arithmetic. Writing a cache entry is not free. The 5-minute write costs 1.25 times the base input rate, and the 1-hour write costs 2 times the base input rate. A cache read costs about a tenth, and the read refreshes the entry at no additional cost every time you use it. So the first request in a session pays full input plus a write premium, and each request after it pays the discounted read.

The lifetime matters more than the rate. The default cache lifetime on Anthropic is 5 minutes, with a 1-hour option at the higher write price above. The lifetime is measured from the start of the request that writes or reads the cache entry, not from the end of its response. If a response takes 4 minutes to stream, a follow-up request that reuses the same prefix must start within about 1 minute of that response completing, or the earlier write expires.

OpenAI runs a different model. Caching is automatic for eligible requests, and there is no cache-write fee on earlier model families. Cache reads are billed at a discounted cached-input rate where a model offers one, and a prompt needs at least 1,024 tokens before caching activates. Newer GPT-5.6 families add explicit cache breakpoints, charge cache writes at 1.25 times the uncached input rate, and use a 30-minute TTL. The retail prices move with model releases, so point your team at the official pricing pages for live figures rather than trusting any number printed here.

Across providers the shape is consistent. Cached input tokens run at a large discount to base input on both OpenAI and Anthropic, roughly an order of magnitude. That discount only applies when the prefix matches, which is why the next section is where the routing question lives.

Why people think routing kills the cache

The objection against stacking sounds reasonable at first. You route each request to the model that fits the task. A cache is keyed to a model and its tokenizer. Switch models and the new one starts cold. It re-processes the whole prompt at the full input rate, and it re-writes the prefix and charges you the write premium. Do that often and your caching wins evaporate.

The kernel of truth is this. Different models cannot share a cache entry, and a model that has not seen a prefix recently will miss. If routing scatters a single workload across many models, each one's cache stays cold because no single model sees enough repeated traffic to keep the entry alive within its short TTL.

That is a real failure mode, but it is a failure of how routing is configured, not a law of combination. The objection assumes routing means churning across models on every request. It does not have to.

Two facts separate the reasonable worry from the myth. First, the cache is keyed on the prefix, and the prefix is the part of the prompt you control to be constant. Routing changes which model computes the tail, not the text of the stable head. Second, the cache lives per model, so the fix is to give each model enough traffic to stay warm rather than to avoid routing entirely.

How a gateway changes the model without breaking the prefix

A gateway sits in front of the provider call. It receives the request, decides which model handles it, and forwards the call. The routing decision and the cache hit operate on different layers of that call, and you keep both by keeping them independent.

The gateway changes one thing: the model field on the outgoing request, and with it the provider and account that carries the billing. The prompt body, and specifically the stable prefix, is data the gateway forwards, not data it invents. If the gateway passes the prompt through untouched, the prefix the provider sees is byte-identical to the one already in that model's cache. The model field changes, the prefix does not, and the cache hit holds.

Model routing selects a destination. Prompt caching matches input text. They touch different fields, and a gateway that keeps those two concerns separate preserves both.

Most gateways make this work through four behaviors.

The gateway passes cache controls through. On Anthropic the request carries cache_control markers or the automatic caching field. On OpenAI it carries the prompt cache key and (on newer models) breakpoints. A gateway must forward those controls verbatim. If it strips them, the provider never knows to cache, and every request pays the full rate regardless of which model answers.

The gateway never rewrites the prefix. Some interception layers inject text into the body, hoist a system message, or reorder blocks for convenience. Any edit before the breakpoint changes the prefix hash and silently kills the hit. The cached region needs to arrive exactly as it was written. The gateway belongs between the client and the provider as a pass-through for the prompt, with routing logic applied at the edges, not inside the cached content.

The gateway routes within a small model set. If every request can land on any of twenty models, no cache stays warm inside a 5-minute TTL. A routing policy that recommends a small set per workload, with fallback only on failure, keeps each chosen model's cache alive. Sticking a session to one model for its lifetime also helps, because a long agentic run keeps re-reading the same growing prefix.

The gateway keeps the account and workspace stable per model. Anthropic isolates cache entries between organizations and between workspaces within an organization. Route traffic for one workload to the same provider account and workspace, and the cache entries are visible to every request in that pool. Scatter a workload across multiple provider accounts and you fragment the cache exactly the way scattering across models does.

Observability closes the loop. The response usage object reports cached tokens, and the gateway should surface that number per request and per model. When the cache read rate on a given model drops, you have changed something about the prompt, the TTL path, or the routing pool. The metric tells you which.

What actually breaks the cache

If routing does not inherently break the cache, something does. These are the concrete causes, and each one maps to a configurable decision.

Editing the prompt upstream. A harness that injects a timestamp, a session id, or a rerouted system message into the static region changes the prefix. So does a gateway that rewrites the system prompt or hoists content for its own reasons. Any of these produces a different hash and a miss.

Reordering before the breakpoint. On Anthropic the cache order is tools, system, then messages. Moving a tool definition after the system prompt, or appending a block to the cached region, changes what the prefix covers.

Broadcasting across too many models. Traffic spread across a wide pool keeps every entry cold inside a short TTL. The remedy is not to abandon routing but to narrow the pool and add per-session stickiness.

Changing parameters that sit inside the prefix. Most sampling parameters operate after the recomputed attention step and do not invalidate a cache. But anything that changes the tokenized input, such as a different tool schema, a different few-shot example, or different image detail settings, is part of the prefix and will miss.

Expiry with no re-write. A cached entry that is never read within its lifetime disappears. Anthropic holds it for 5 minutes by default or 1 hour at the higher write price; OpenAI evicts earlier models after minutes of inactivity and newer models at 30 minutes. Low-traffic models simply have no entry to hit. This is a routing-pool problem, not a routing-in-general problem.

How providers actually behave in practice

The gateway design above is provider-agnostic, but the two big providers handle the routing-to-cache dynamic differently, and knowing the difference sets expectations.

Anthropic gives you explicit control. You mark the breakpoints, you choose the TTL, and you pay the write fee for the privilege. In the author's testing against a long prompt with every input token cached, requests landed on cached entries essentially every time caching was requested, with a substantial reduction in time-to-first-token. The trade is that you must structure and mark the prompt correctly, and you pay for writes.

OpenAI does caching automatically for eligible requests. The provider routes a request to a server that holds a matching prefix and applies the discounted read. In the same testing, resending a request immediately produced a hit rate around half, and the author notes that variability can show up as inconsistent latency on long context windows. The newer model families add explicit breakpoints and a named TTL for teams that want determinism.

The takeaway for a gateway operator is that Anthropic's model rewards deliberate setup, while OpenAI's model rewards steady volume on a stable prefix, because the hit rate is the lever you live with either way. Either way the cached prefix must stay exact, and concentrated traffic raises the hit rate.

When stacking the two pays off

The rule of thumb across practitioners is that caching pays when a large share of your input repeats across requests. One widely cited threshold puts it around half of input tokens repeating. Below that, the write fee and the short TTL can make caching neutral or worse. Above it, the discount on the repeated bulk dominates.

Three workloads sit comfortably above the line. Agentic coding loads the same codebase context and tool definitions into every turn. RAG assistants inject the same reference blocks while only the question changes. Long multi-turn support bots append new messages to a large, stable conversation history. All three send a big, constant prefix and a small, variable tail, which is the exact shape caching rewards and routing can exploit.

Routing adds value on top because these workloads do not all need the same model. Classification and extraction inside a support flow run fine on an entry tier. Open-ended reasoning benefits from a frontier tier. When the stable prefix is cached, the cost difference between sending that prefix to a less expensive model and a frontier one narrows to the read rate on each, while the difference in the fresh tail and the output stays meaningful. You route to save on the parts that are not cached, and you cache to keep the shared parts inexpensive on whichever model you reach.

Worked example: a support assistant with a long cached prefix

The numbers below are assumptions, labeled so you can swap in your own token counts. The point is the shape, not a quote.

Take an assistant that injects 80,000 tokens of product docs and tool definitions into every request. Those docs never change, so they form the cached prefix. Each user turn adds roughly 2,000 tokens of fresh input and produces about 1,500 tokens of output. The mid tier charges $3 per million base input, $0.30 per million cache read, and $15 per million output.

On the first request of a session, the provider writes the 80,000-token prefix to cache. That write bills 80,000 tokens at the 5-minute write rate of $3.75 per million, about $0.30. A request that hits the cache instead reads those same 80,000 tokens at $0.30 per million, about $0.02 each, which is a tenth of the uncached $0.24 bill for that bulk.

Ten requests in a session with the same model look like this. One write at $0.30, plus nine reads at $0.02, puts the cached prefix at about $0.48. Uncached, those same ten requests re-process the 80,000-token bulk at the full rate, about $2.40. The cache holds the savings inside the session so long as the requests arrive within the TTL.

Now route half of those requests to an entry tier that also holds the prefix. The entry tier reads cached input at $0.10 per million, and the write costs $1.25 per million, so the same arithmetic applies at a lower absolute level on the routed share. What matters is not that the models differ but that each model keeps seeing the prefix often enough to hold its own cache entry within the lifetime. Two models sharing a stable prefix, with steady traffic to each, both stay warm. Twenty models sharing that prefix get none of the discount.

A checklist for stacking routing and caching

Apply these in order and the pair works together.

Structure the prompt with the stable prefix first. System prompt, tools, schemas, examples, and reference context at the front. User-specific content after.

Mark the cache boundary. Use explicit breakpoints on Anthropic or automatic caching, and rely on the 1,024-token minimum on OpenAI. Confirm the write and read pricing for your model tier.

Pass cache controls through the gateway. Forward cache_control, prompt cache keys, and breakpoints verbatim.

Refuse prompt rewrites in the pipeline. Strip any layer that injects or reorders content before the breakpoint.

Route within a small set of models per workload. Hold sessions to one model where you can. Keep each model's cache warm with steady traffic.

Keep one provider account and workspace per workload. Cache entries are isolated across organizations and workspaces.

Watch the cached-token metric. If the read rate drops, trace the change to a prompt edit, a TTL miss, or a widened routing pool.

Frequently asked questions

Does model routing break the cache? Not by itself. Routing changes which model handles the request. Caching matches the prompt prefix inside that model. Send the same stable prefix to every request and route within a model set that stays warm, and the cache hits hold.

Can a gateway switch models mid-session and keep the cache? Yes, if both models have seen the prefix recently. Each model keeps its own entry. The first call to a newly chosen model writes its cache, and later calls read it. Avoid switching to a model the session has never used, because that model starts cold and bills the full write.

Does a different temperature or top-p invalidate the cache? No. Sampling parameters act after the attention computation that prompt caching reuses, so you can change them freely without invalidating cached input.

How long does the cache last? On Anthropic the default lifetime is 5 minutes and the extended option is 1 hour, at 2 times the base write price. On OpenAI, newer model families use a 30-minute TTL and earlier models evict after minutes of inactivity. Each successful read refreshes the entry.

What is the fastest reliable way to make routing pay? Bypass the question of savings percentages and look at the mechanism. Cache the repeated prefix at the discounted read rate, route only the non-cached tail and the output toward the model that fits the task, and concentrate traffic on a small model set so the discounts actually land.

The takeaway

Prompt caching and model routing answer two separate questions, and that is why they stack. Caching decides how you pay for the repeated prefix. Routing decides who computes the part that changes. The gateway that keeps those concerns apart, by passing the stable prefix through untouched and routing within a warm model set, keeps both benefits at once.

The discipline that makes either one work is the same discipline that shows up across the whole cost problem. A stable prefix that you structure once and stop editing. A routing policy that favors a small set of models. A cached-token metric that tells you when something drifted. The pricing that anchors this whole discussion, and the levers that cut it, live in the Claude Code cost guide, while the broader LLM cost optimization playbook sets caching and routing inside the full sequence of nine tactics.

Revisit your prompt layout first. If the static content does not lead, no routing policy or gateway setting can save you. Fix the order, turn on caching at the boundary, and route within a pool that stays warm, and the objection that routing kills the cache stops being true for your workloads.