Prompt Caching in Provider-Agnostic Agents

Suppose you are working on a feature with a coding agent. Your thread amounts to 100k tokens. You send a message to your agent with 1k tokens. How much are you charged for this message? 101k tokens at full price?

Not if your agent has configured prompt caching correctly.

Prompt caching is a practice that allows your inference providers to cache the processing of a part of your prompt. This allows them to save resources, which in turn reduces costs and latency for your agent.

In this writeup, we will explain how to do it right when designing your own agent.

Benefits of Prompt Caching

Whenever prompt-caching is available, it comes with:

• Lower TTFT (Time To First Token)
• Lower latency in general
• Lower cost per cached token

Coding agents are especially token-hungry. Caching allows you to have long conversations with them at a discounted price. How much of a discounted price is that? — A lot.

Across the major inference providers, the caching multipliers look like this (relative to the base input price of their flagship coding model):

Different providers make you manage cache differently. For example, OpenAI’s models above GPT-5.6 will automatically write the cache on first request, and you’ll need to explicitly disable it if you don’t want it. You can also add an optional session ID to your requests to make it easier for OpenAI to route your request to the instance with the warmest cache. On the other hand, the cache is disabled by default on providers like Google and Anthropic, and you need to request it on your API calls.

The cache policy may also change depending on the endpoint you are using (even for the same LLM). For example, Anthropic’s default cache TTL for API calls is 5 minutes, and for calls to the OAuth endpoint (like Claude Code) it is 1 hour.

So check your providers to make sure that you are optimizing costs correctly for every provider that you support on your agent.

What a Long Session Actually Costs

Multipliers are abstract. Let’s make this data concrete with a realistic coding-agent session.

Take a session that grows to ~200k tokens: a 10k system prompt with tool definitions, then user messages of 1k tokens each. For every user message, the agent performs 10 tool calls. Each tool cycle adds roughly 400 output tokens (the tool call plus some reasoning) and a 2.5k-token tool result — think of reading a ~200-line file. That makes each full agent turn worth about 30k tokens, so our session spans about 7 user turns.

The important detail: every tool call is a new request that re-submits the entire conversation prefix. Without caching, you pay full input price for a prefix that grows from 10k to 220k tokens, 77 times in a row. With caching, you pay the full price (or a small write premium) only for the new tokens, and read the rest at the discounted hit price. The exact per-provider rates driving the plots below are listed in the footnotes.⁷

The plot shows the total cost of the full session. A few things stand out:

• Caching is worth 5–7x on every provider. On Anthropic and OpenAI, the session drops from ~$42 to ~$6. On DeepSeek, it drops from $3.58 to $0.14 — a 96% discount, thanks to its nearly-free cache hits.
• The write premium barely matters. You may be tempted to think that writing the cache is too expensive. OpenAI charges 1.25x for cache writes¹, Anthropic can charge up to 2x for a 1-hour-TTL cache write ². But you write each token once and read it many times. It’s still much cheaper to write the cache in long conversations.

A coding agent without prompt caching will re-read your conversation at full price.

One caveat: several providers charge more once a single request crosses a long-context threshold (typically 200k tokens)⁸, and Google bills cache storage per hour³. The plot uses short-context tier prices and includes Gemini’s cache storage assuming a one-hour session, so treat the last calls of the session as a slight underestimate. The conclusion holds either way.

Watching the Cost Accumulate

Here is the same example (200k token session), plotted as cumulative cost against the size of the conversation. Solid lines are with caching, dotted lines without. Use the checkboxes to filter providers, and hover over the lines to inspect values:

The dotted lines bend upward: every new tool call re-reads a longer prefix, so each step costs more than the previous one. The cost of a non-cached conversation increases quadratically — the longer the session, the worse it gets.

The solid lines tell the opposite story. They stay nearly flat, because each step only pays full price for the few thousand new tokens and reads the rest at a 10x (or, for DeepSeek, 120x) discount. DeepSeek’s line is so flat it is practically glued to the x-axis.

Without caching, long sessions get exponentially expensive.

The Golden Rule: Stable Prefixes

A cache hit happens only when your prompt matches a previous prompt token-for-token, starting from the very first token. Change a single token at position i, and all your cache is invalidated.

Prompt caching does not forgive edits.

• Order by stability. If you’re not building a conversation agent (e.g. 1-off requests with a system prompt), make a stable prefix and cache it. Keep the most stable content first: system prompt, then tool definitions, then conversation history, then the newest tool results. Volatile content always goes last.
• No surprises at the top. No timestamps, request IDs, random seeds or user metadata injected at the beginning of the system prompt. One changing token there invalidates the entire cache on every call.
• Append-only history. Never rewrite old messages. If you must change something, accept that everything after it will be re-processed at full (or write) price.

What Breaks Your Cache

In practice, this means that most agents hit the cache naturally when you keep sending back the same transcript or when you’re using the Responses API. But there are a few common pitfalls:

• Dynamic tool lists. MCP servers that connect mid-session, or tools that are loaded lazily, change the tool definitions — and with them, the prefix.
• Mid-conversation system prompt edits. Dynamically including in your system prompt things like current time, git branch, open files, etc. is a great way to invalidate your cache.
• Context compaction. Compaction works by replacing your thread with a summary to reset the context window. This doesn’t mean that you shouldn’t do compaction, but do expect one full-price call at the start of a compacted session.
• TTL expiry. This is another common pitfall. And one that I may explore in a future post. Sometimes, your user will go grab a coffee in the middle of a task, or will resume a thread after a few days. When your user gets back, the cache will have already expired. This is what happened in the figure below.
• Non-deterministic serialization. Unordered JSON keys in tool schemas, floats formatted differently, a set iterated in hash order. If your prompt is not byte-stable across processes, your cache is not either.

Key Takeaways

• Prompt caching gives you 5–7x cheaper long agent sessions on every major provider — and up to 25x on DeepSeek.
• Caches match prefixes. Order your prompt by stability, append-only, nothing volatile at the top.
• “Automatic” providers still need an affinity key and a stable prefix; explicit providers need both plus correct annotations.
• Retention is an economics decision. Tie the TTL to your auth mode, not to a constant.
• Capability should narrow intent through per-route flags, never a hardcoded provider name.
• Compaction, dynamic tools, wide parallel turns and TTL expiry will break your cache. Plan for them.
• Track your hit rate — latest and cumulative. A cache that works while its metrics lie is a cache you will eventually break without noticing.

A well-cached agent doesn’t just cost less. It answers faster — and that’s the feature your users actually notice.