Prompt Caching in Provider-Agnostic Agents

Suppose you are working on a feature with a coding agent. Your thread amounts to 100k tokens. You send a message to your agent with 1k tokens. How much are you charged for this message? 101k tokens at full price?

Not if your agent has configured prompt caching correctly.

Prompt caching is a practice that allows your inference providers to cache the processing of a part of your prompt. This allows them to save resources, which in turn reduces costs and latency for your agent.

In this writeup, we will explain how to do it right when designing your own agent.

Benefits of Prompt Caching

Whenever prompt-caching is available, it comes with:

• Lower TTFT (Time To First Token)
• Lower latency in general
• Lower cost per cached token

Coding agents are especially token-hungry. Caching allows you to have long conversations with them at a discounted price. How much of a discounted price is that? — A lot.

Across the major inference providers, the caching multipliers look like this (relative to the base input price of their flagship coding model):

Two axes are worth separating here, because they are easy to conflate. The first is who decides what gets cached: Anthropic and Google’s explicit cache make you ask for it, everyone else caches prefixes automatically. The second is whether writes cost extra: Anthropic charges a write premium², OpenAI’s newer models bill first-time prefix processing above the base input rate¹ (which is why their usage payload reports a cache-write count at all), and Kimi, xAI and DeepSeek charge nothing extra — for them every request is simply a potential hit.⁴⁵ DeepSeek is the outlier on reads: its cache hits are almost free, at less than 1% of the base input price.⁶

So “automatic” does not mean “free”, and “explicit” does not mean “expensive”. The write premium is a one-time toll on tokens you are about to read many times over.

Different Caching Mechanisms

Not all caches work the same way. There are three mechanisms in the wild, and your agent design has to respect whichever one your provider uses.

Automatic prefix caching (OpenAI, DeepSeek, Kimi, xAI). The provider caches processed prompt prefixes and automatically serves a hit when your next request shares a prefix with a recent one. There is no cache object to manage and no breakpoint to place — but “automatic” is not the same as “nothing to do”:

• Caching only kicks in above a minimum prefix (1024 tokens on OpenAI) and matches in coarse increments (128 tokens), so the tail of your prompt is never the part that hits.⁷
• Requests are routed to whichever machine holds a warm cache, and OpenAI lets you steer that routing with an optional prompt_cache_key. Same prefix, wrong machine, cold cache. This is the one knob “automatic” providers do give you, and agents should use it.⁷
• Retention is short and not contractual: roughly minutes of inactivity, longer during off-peak, with no TTL you control (OpenAI exposes an extended-retention opt-in on some routes).⁷

Explicit breakpoints (Anthropic). You mark positions in your prompt with cache_control: {"type": "ephemeral"}, and everything up to and including that block becomes a cacheable prefix. Details that matter when you implement it:

• Up to 4 breakpoints per request; a fifth is a hard 400.⁸
• The prefix is evaluated in a fixed order: tools → system → messages. Your breakpoints have to respect that order.⁸
• Default TTL is 5 minutes, refreshed on every hit. A 1-hour TTL is available to any caller — API key included — by adding "ttl": "1h" to the cache_control block, and costs 2x the base input rate on writes instead of 1.25x. Reads stay at 0.1x either way.²
• Minimum cacheable prefix is 1024 tokens for the large models (2048 for the small ones). Shorter prefixes are silently not cached.⁸
• On a read, Anthropic only looks back a bounded number of content blocks (~20 positions) from each breakpoint before giving up. A single agent turn with a lot of parallel tool calls can append more blocks than that — which is why one breakpoint at the tail is not always enough. More on this below.⁸
• Anthropic has no automatic caching: no cache_control, no cache. This is the trap. An agent that never writes a breakpoint re-bills its entire system prompt, tool schema and transcript on every single tool call, and nothing in the response tells you unless you read the usage fields.

Explicit cache objects (Gemini). Google runs both modes. Implicit caching is on by default on recent models and behaves like the automatic prefix caches above — discounted hits, no storage fee, no control. Explicit caching has you create a cache resource through the API — a named object holding your system prompt and tool definitions — choose a TTL, and reference it in subsequent requests. Explicit buys you a guaranteed hit and a TTL you control, and charges storage per million tokens per hour on top of the discounted reads.⁹ It is also the only mechanism here that requires a separate API call and a lifecycle to manage, which is exactly why most agents skip it and lean on implicit caching.

The mechanisms differ. The rule they all obey is the same: caches match prefixes. Which brings us to the most important design principle of this writeup.

What a Long Session Actually Costs

Multipliers are abstract. Let’s make them concrete with a realistic coding-agent session.

Take a session that grows to ~200k tokens: a 10k system prompt with tool definitions, then user messages of 1k tokens each. For every user message, the agent performs 10 tool calls. Each tool cycle adds roughly 400 output tokens (the tool call plus some reasoning) and a 2.5k-token tool result — think of reading a ~200-line file. That makes each full agent turn worth about 30k tokens, so our session spans about 7 user turns.

The important detail: every tool call is a new request that re-submits the entire conversation prefix. Without caching, you pay full input price for a prefix that grows from 10k to 220k tokens, 77 times in a row. With caching, you pay the full price (or a small write premium) only for the new tokens, and read the rest at the discounted hit price. The exact per-provider rates driving the plots below are listed in the footnotes.¹⁰

The plot shows the total cost of the full session. A few things stand out:

• Caching is worth 5–7x on every provider. On Anthropic and OpenAI, the session drops from ~$42 to ~$6. On DeepSeek, it drops from $3.58 to $0.14 — a 96% discount, thanks to its nearly-free cache hits.
• The write premium barely matters. Anthropic and OpenAI charge 1.25x for cache writes²¹, but you write each token once and read it many times. The hit discount dominates the math.
• Cached prices compress the field. Without caching, the most expensive session costs 12x the cheapest one. With caching, the gap narrows — the expensive part of a long agentic session is exactly the part caching discounts away.

A coding agent without prompt caching is a subscription to re-reading your own conversation at full price.

One caveat: several providers charge more once a single request crosses a long-context threshold (typically 200k tokens)¹¹, and Google bills cache storage per hour³. The plot uses short-context tier prices and includes Gemini’s cache storage assuming a one-hour session, so treat the last calls of the session as a slight underestimate. The conclusion holds either way.

Watching the Cost Accumulate

Totals hide the shape of the problem. Here is the same simulation, plotted as cumulative cost against the size of the conversation. Solid lines are with caching, dotted lines without. Use the checkboxes to filter providers, and hover over the lines to inspect values:

The dotted lines bend upward: every new tool call re-reads a longer prefix, so each step costs more than the previous one. That is the quadratic tax of running an agent without caching — the longer the session, the worse it gets.

The solid lines tell the opposite story. They stay nearly flat, because each step only pays full price for the few thousand new tokens and reads the rest at a 10x (or, for DeepSeek, 120x) discount. DeepSeek’s line is so flat it is practically glued to the x-axis.

Without caching, long sessions get expensive. With caching, they just get long.

The Golden Rule: Stable Prefixes

A cache hit happens only when your prompt matches a previous prompt token-for-token, starting from the very first token. Change a single token at position i, and everything after i is a cache miss — even if 99% of the prompt is identical.

Prompt caching does not forgive edits. It only forgives appends.

This has concrete consequences for how you assemble requests:

• Order by stability. Most stable content first: system prompt, then tool definitions, then conversation history, then the newest tool results. Volatile content always goes last.
• No surprises at the top. No timestamps, request IDs, random seeds or user metadata injected at the beginning of the system prompt. One changing token there invalidates the entire cache on every call.
• Append-only history. Never rewrite old messages. If you must change something, accept that everything after it will be re-processed at full (or write) price.

What Breaks Your Cache

Coding agents have a few specific habits that silently destroy hit rates:

• Dynamic tool lists. MCP servers that connect mid-session, or tools that are loaded lazily, change the tool definitions — and with them, the prefix. If you can, fix the tool list at session start. If you can’t, append new tools at the end rather than inserting them, so the invalidation is partial. (Kimi K3’s dynamic tool loading is designed exactly around this problem.)
• Mid-conversation system prompt edits. Updating the system prompt to reflect new state (current time, git branch, open files) feels cheap. It costs you the whole cache.
• Context compaction. When your agent summarizes and rewrites the history to fit the context window, the new history is a brand-new prefix. Expect one expensive turn where everything is re-written to the cache. It is still worth it — just don’t be surprised by the spike.
• TTL expiry. Anthropic’s 5-minute cache dies while your user reads the diff, goes for coffee, or thinks. Every request refreshes it, but a slow conversation keeps paying write prices. This is what the 1-hour TTL is for. In practice, on a well-built agent, idle gaps are the dominant cause of misses — not bad breakpoint placement.
• Wide turns outrunning the lookback window. On Anthropic, a request with many parallel tool calls appends one tool_use block per call plus one tool_result block per result. Cross the ~20-block lookback and your previous cache entry is technically still alive but no longer findable from the breakpoint you placed. The fix is a second breakpoint further back, not a bigger cache.
• Cache locality. Caches live at your provider, per account or organization, often per machine or region. A retry that lands elsewhere gets a cold cache — which is what affinity keys like OpenAI’s prompt_cache_key exist to prevent.
• Non-deterministic serialization. Unordered JSON keys in tool schemas, floats formatted differently, a set iterated in hash order. If your prompt is not byte-stable across processes, your cache is not either.

A Provider-Agnostic Caching Layer

How do you support all of this without forking your agent per provider? The trick is to separate two concerns: keeping the prefix stable (your agent’s job, always) and translating cache hints (a thin adapter’s job, per provider).

Keep your context as an ordered, append-only list — system prompt, tools, history — and let a prepare_request function handle the rest:

def prepare_request(provider, system, tools, messages, session_id):
    if provider == "anthropic":
        # Explicit breakpoints, spent within a budget of 4:
        #   1. last tool schema  -> caches the whole tool block
        #   2. final system block
        #   3. previous request's tail  -> second lookback window
        #   4. this request's tail      -> extends the cache to the tip
        ttl = "1h" if is_subscription_auth else "5m"
        return anthropic_request(
            system=mark_last_block(as_blocks(system), ttl),
            tools=mark_last_tool(tools, ttl),
            messages=mark_request_tails(messages, ttl),
        )

    if provider == "gemini":
        # Explicit cache object: create once per session, reuse after.
        # Skipping this is legitimate: implicit caching still applies.
        cache = ensure_cache(system, tools, ttl="1h")
        return gemini_request(cached_content=cache.name,
                              messages=messages)

    # openai, deepseek, kimi, xai: automatic prefix caching.
    # No breakpoints to place - but do hand over a stable affinity
    # key so successive calls route to the same warm cache.
    return openai_request(
        system=system,
        tools=tools,
        messages=messages,
        prompt_cache_key=session_id,  # OpenAI only; gateways may reject it
    )

Notice what the abstraction does not do: it does not reorder, deduplicate or “optimize” your messages. That discipline lives in the agent loop itself. The adapter only encodes, for each provider, where the stable prefix ends and how to ask for it.

Two things are worth pulling out of that snippet, because they are the parts people get wrong:

Retention is an economics decision, not a technical one. Anthropic sells the 1-hour TTL to anyone, so this is never a question of capability — only of whether the premium pays off. A 1-hour write costs +0.75x input over a 5-minute one; letting the cache expire and re-writing costs +1.15x over reading it back. So the longer TTL wins if the chance of at least one >5-minute idle gap mid-session is above roughly 65% — nearly always true for interactive work, nearly never for a scripted batch run. On a subscription the premium isn’t billed per token at all, so it is simply free.

Which means the TTL should follow the auth mode rather than be hardcoded either way: ask for an hour where it costs nothing, keep the short default where a user would be silently charged for a bet the agent can’t make on their behalf.

“Anthropic-compatible” is not Anthropic. Plenty of gateways speak the Anthropic wire protocol without implementing cache_control, and some proxy to non-Anthropic models entirely. Sending breakpoints there ranges from silently useless to a hard 400. Capability therefore has to be a per-route flag that can only ever narrow what you intended to send — never a hardcoded if provider == "anthropic".

Measuring Your Cache Hit Rate

You cannot fix what you don’t measure. Every provider reports cache usage in the response metadata:

Two traps in that table. First, the field names differ between OpenAI’s two APIs — Responses nests under input_tokens_details, Chat Completions under prompt_tokens_details — so an adapter that only reads one will silently report a 0% hit rate on the other. Second, Anthropic’s cache_creation_input_tokens is a total that already includes the 1-hour writes reported separately in cache_creation.ephemeral_1h_input_tokens. Bill the whole amount at the 5-minute rate and you understate the cost of every 1-hour session; add the two together and you double-count.

Compute your hit rate as cached tokens over total input tokens, per request. For a coding agent in a steady loop, anything above 90% is healthy — only the newest tool results should miss.

It is also worth tracking two rates rather than one. A cumulative session rate is the billing-relevant number, but it is dragged down forever by the unavoidably cold first request — so it is useless as a diagnostic. The rate of the latest request tells you whether caching is working right now, which is the thing you actually want to see after changing a prompt.

More importantly: alert on drops. A sudden fall in hit rate almost always means one of the cache-breakers above — a tool list changed, a timestamp sneaked into the system prompt, a compaction just ran. The hit rate is your unit test for prefix stability.

Case Study: How This Was Implemented in Tau

All of the above is theory until you have to ship it across a catalog of providers that disagree with each other. Tau is a provider-agnostic coding agent, so it is a decent worked example — including the mistakes.

The starting point is the best illustration of why measurement matters. Tau read cache usage from the very first release: cache_read, cache_write and the 1-hour subset were all parsed off Anthropic’s message_start event and priced. But it never placed a single cache_control breakpoint in a request. The reporting half was wired up and the requesting half was not, so those counters sat at zero on every real turn while each request re-billed the system prompt, the whole tool schema block and the entire conversation as fresh input. Nothing failed. Nothing warned. It was noticed because a subscription’s usage limit was being consumed several times faster than the equivalent work in Claude Code. After the fix, long sessions report hit rates in the high 90s.

Anthropic: spending four breakpoints

The budget is four markers, evaluated in tools → system → messages order. Tau spends them like this:

Breakpoint 2 is a small lesson in not wasting slots. Under subscription OAuth the system field holds an identity block followed by Tau’s own prompt. Marking the identity block would cache a ~15-token prefix that the block after it already covers — a marker spent on nothing.

Breakpoints 3 and 4 are the interesting pair, and they exist because of the lookback window described earlier. One agent turn appends 2N+2 blocks for N tool calls, so a turn with nine or more parallel tool calls pushes the previous cache entry outside the ~20-position window and misses — the cache is alive, just unreachable from a single tail marker. Marking where the previous request ended opens a second lookback window closer to the reusable prefix.

The implementation detail I like most: that second position is not remembered across requests. The transcript is append-only and every request stops immediately before the assistant message it produces, so the previous request’s tail is exactly the last user-role message preceding the final assistant message — recoverable from the payload alone, with no state to keep in sync. It is also worth noting that “request” is not “user turn”: all four markers are recomputed on every tool round trip, and because Anthropic sends tool results with role: "user", these breakpoints land on tool_result blocks far more often than on anything a human typed.

Measured against the live API:

The third row is the case breakpoint 3 exists for: 26 new blocks is outside the lookback, and a single-breakpoint design misses there.

Retention follows the auth mode

Tau resolves retention as one of none, short or long, and — per the economics argument above — derives it from how you are authenticated:

• Subscription OAuth → long (ttl: "1h"). Tokens are not billed per unit, and 5 minutes is shorter than a test run, a build, or reading a diff.
• API key → short, the provider default, so nobody silently pays the 2x write premium they did not ask for.²
• Anthropic-protocol gateways → none, leaving the payload byte-identical to the pre-caching shape.

Crucially, intent and capability are resolved separately. Intent comes from the auth mode; capability comes from three catalog flags layered detected → provider → per-model:

Capability only ever narrows intent, so the two compose with no precedence rule — and the failure mode is recoverable without a code change. An unsupported ttl is a hard 400, and Tau does not retry 400s, so if a provider ever stops honoring one-hour retention a three-line catalog overlay clamps it back.

OpenAI: affinity instead of breakpoints

There is nothing to mark on the OpenAI family, so the work is entirely about routing. Tau carries its durable coding-session ID through the provider-neutral request boundary — including tool continuations — and uses it as prompt_cache_key, clamped to OpenAI’s 64-character limit. Direct Responses requests also send it as a session_id header; Codex OAuth spells the same header session-id; Chat Completions sends the body field only.

A few deliberate non-decisions there, all of which are easy to get wrong:

• x-client-request-id is not used for this. OpenAI defines it as a unique per-request diagnostic ID; reusing a session ID there makes your own provider logs ambiguous.
• Requests stay stateless (store: false) and keep sending the full transcript. An affinity key improves cache grouping; it cannot make a changed prefix cacheable.
• Compaction and summary requests deliberately omit the affinity key, because their one-off prompts share no prefix with the conversation.
• Compatible gateways keep the old request shape unless a catalog flag opts them in — same narrowing principle as the Anthropic flags.

The parts that were wrong for a while

Two follow-ups are worth naming, because they are the kind of thing that hides for months:

Cache-write accounting on the OpenAI side initially treated write tokens as ordinary fresh input, because the Responses API reports them under a different key than Chat Completions. Fixing it meant parsing input_tokens_details.cache_write_tokens and preserving the invariant that input + cache_read + cache_write reconstructs the provider’s reported total.

1-hour writes were billed at the 5-minute rate. Anthropic’s cache_creation_input_tokens already includes the 1-hour writes reported separately, and every pricing consumer billed the whole amount at the 1.25x rate while OAuth sessions default to the 2x tier.² The result: subscription sessions — exactly the ones producing 1-hour writes — systematically understated cost by roughly a third on the write portion. The fix was a separate cacheWrite1h catalog rate with a fallback to cacheWrite when a model has no such entry, so gateway and user-supplied catalogs bill exactly as before.¹³

Both bugs share a shape: the cache worked, and only the numbers reporting on it were wrong. Which is the whole argument for treating cache metrics as a first-class part of the agent rather than a footnote.

What it looks like to the user

Tau’s sidebar shows two rates — the latest model request and the cumulative session — and hides both entirely when no provider in the session has reported any cache activity, rather than displaying a permanently misleading 0%. That distinction is the difference between “caching is broken” and “this backend doesn’t report it”, and users cannot tell them apart on their own.

One known gap, since honesty is cheaper than a surprise: compaction and branch-summary requests still inherit full retention, so they write a cache entry that can never be read back — they use their own system prompt and no tools, sharing no prefix with the conversation. Cheap to leave, easy to fix, not yet fixed.

Key Takeaways

• Prompt caching gives you 5–7x cheaper long agent sessions on every major provider — and up to 25x on DeepSeek.
• Caches match prefixes. Order your prompt by stability, append-only, nothing volatile at the top.
• “Automatic” providers still need an affinity key and a stable prefix; explicit providers need both plus correct annotations.
• Anthropic has no automatic caching at all. No breakpoint, no cache, no warning.
• Retention is an economics decision. Tie the TTL to your auth mode, not to a constant.
• Capability should narrow intent through per-route flags, never a hardcoded provider name.
• Compaction, dynamic tools, wide parallel turns and TTL expiry will break your cache. Plan for them.
• Track your hit rate — latest and cumulative. A cache that works while its metrics lie is a cache you will eventually break without noticing.

A well-cached agent doesn’t just cost less. It answers faster — and that’s the feature your users actually notice.