Prompt caching for agents

The rule

The cache matches a prefix: the longest run of tokens from the start of your prompt that is byte-identical to a previous request. Everything from the first difference onward is recomputed and billed as uncached.

So an agent turn that appends to the end of its context reuses almost everything. A turn that changes something near the beginning reuses almost nothing, however small the change.

The match is counted in whole cache blocks, and the trailing partial block is billed at the input rate. The unit size varies by model, so a very short prompt may never hit, and the discount does the most work on long prefixes.

One more shape matters on some models: a conversation that appends warms on turn 2, but a shared prefix re-sent with a changing suffix, such as many questions against one document, may take an extra send before cached_tokens shows it warm. The model-specific caching reference identifies the model for this observed behavior.

What breaks a prefix

Every item below has been observed reducing a real workload's hit rate. They are ordered by how often they turn out to be the cause.

Reading your own hit rate

Chat Completions responses report usage.prompt_tokens_details.cached_tokens, the count of input tokens that request was billed at the cached input rate, and the same number as usage.runinfra.cached_input_tokens beside the cost. It is 0 rather than absent when nothing was billed as cached. The Responses API reports it as usage.input_tokens_details.cached_tokens, and Messages as usage.cache_read_input_tokens. Print it on every turn: on a workload that should be repetitive, the value that keeps coming back is where your prompt stops matching itself. The field is described in Chat completions.

Your workspace's cache report is on the dashboard's Usage Analytics page, read with your signed-in dashboard session rather than an API key. The Prompt cache band carries the workspace totals: cached input tokens, the confirmed hit rate, confirmed misses, and confirmed savings. The By API key and By model tables on the same tab carry cached tokens, hit rate, and savings per key and per model. A low rate on a workload that should be repetitive means something above is changing.

To compare many requests, record their cached-token counts and look at the distribution, not just an average. Its most useful summary is the mode: the cached-token count that recurs most often. Because matching is counted in whole cache blocks, this is roughly where your prompt stops matching itself; past that point, a field in your prompt may be differing between requests.

response = client.chat.completions.create(
    model="MODEL_ID", messages=messages, max_tokens=16384
)
print(response.usage.prompt_tokens_details.cached_tokens)

If the count that recurs sits far below your prompt size, look at what is between that offset and the start of your context.

A worked example from real traffic, measured in August 2026: one workload showed a mode of 9,984 cached tokens on a fifth of its requests, against prompts averaging about 70,000 tokens. Everything up to roughly token 10,000 was reusable and nothing after it was, on every affected turn. That is the signature of a field changing early in the prompt, not of a cache problem. A sibling workload on the same model reused 92 percent of a 465,000-token prompt. These are measured examples, not a promised hit rate.

Keeping a session warm

Check current model availability in GET /v1/models. The retention behavior below describes configured models and does not guarantee that a model is serving.

On every chat model, retention of a warm prefix is best effort: an idle prefix can be evicted under capacity pressure, and no lifetime is promised.

On /v1/models, cache_tiers and cache_retention tell you which kind of cache a model runs. best_effort with cache_tiers ["gpu"] is the GPU copy alone. session_affinity_idle_ttl_seconds is a routing window, how long a quiet session keeps its place close to its cache, not a retention promise.

tiered_host_memory means a prefix can outlive the GPU tier; retention there is still best effort.

The cached-token count on the first turn after a pause is the readout of what survived it. If the copy was lost while your agent ran tools, waited on a human, or slept overnight, that turn is billed as a miss; the count and the bill agree. The matching rule at the top of this page is unchanged: a returning prompt must be byte-identical to the prefix it wants back.

On any chat model, a stable x-session-id keeps a session's follow-up turns close to its warm cache, and pins that placement for the model's published session_affinity_idle_ttl_seconds idle window. Without one, a prompt_cache_key in the request body routes the request by its value but does not pin it. Without either, placement still keeps steady sessions warm; the full placement order is on the session affinity page.

What does not affect caching

• Sampling parameters. temperature, top_p, seed, and max_tokens are not part of the prefix.
• Streaming. A streamed and a non-streamed request cache identically.
• Your traffic volume. Caching is per prefix, not a plan or an allowance.

Isolation

On models that publish cache_isolation: "isolated" in /v1/models, a cached prefix is scoped to your workspace within the model's published isolation scope. Another customer does not reuse your isolated prefix, and you do not reuse theirs. That scoping applies on every request. Read the per-tier isolation scope in the cache isolation reference. On a model that does not publish isolation, a hit is neither disclosed nor priced. A billable response charges every input token at the standard input rate. The response reports cached_tokens: 0.

Next steps