中文

Prefix Stability Across Four Prompt-Cache Mechanisms

·Codot Engineering

An agent runs as a loop of model calls. At each step the full context is sent again (instructions, tool schema, message history, tool results), and the model decodes a short structured output. The workload is therefore dominated by prefill: input tokens exceed output tokens by orders of magnitude. For such a system, both cost and time-to-first-token are determined largely by how much of that input the serving side can restore from a key-value (KV) cache instead of recomputing.

Prompt caching refers to the reuse of this precomputed state across requests. Reuse is prefix-based: a request can restore state only for the span of tokens that is identical to an earlier request from the first position, and the first differing token invalidates everything after it. Improving cache reuse therefore reduces to a single engineering problem: keeping the longest possible prefix byte-identical across the set of requests that are able to share it.

What is less often discussed is that this set of requests is not the same for every provider. We run several models in production, and across them we have encountered four distinct cache mechanisms, each with a different scope of reuse. In this post we classify those mechanisms, describe how we located the point at which our own prefix was being invalidated, and share the prompt layout and routing we now use for each.

Four cache mechanisms

Cache mechanisms can be distinguished along three dimensions: who creates a cache entry, where the entry is stored, and what a request must satisfy in order to hit it.

MechanismEntry created byStorageHit condition
Explicit cache objectThe caller, through a separate call with a chosen time-to-live (TTL)A named, addressable objectThe request references the object before its TTL expires
Implicit prefix cacheThe provider, at its discretionUndocumentedPrefix equality with an earlier request, on a best-effort basis
Global automatic prefix cacheThe provider, on every requestShared, long-lived storagePrefix equality with an earlier request from the first position
Per-machine automatic prefix cacheThe provider, on every requestThe memory of a single inference nodePrefix equality, and routing to the same node

The first two are alternatives offered for the same model: the caller either constructs the cache explicitly or relies on the provider to cache what it can. We evaluated both on the same traffic. The implicit mode produced a substantially lower hit rate and a higher cost per turn, so we use the explicit mode.

Scopes of reuse

A cache entry is only valuable if a later request reuses it. For an agent, we find it useful to distinguish four nested scopes in which reuse can occur.

ScopeDefinition
Cross-userThe portion of the prompt that is identical for every user: instructions and tool schema
Cross-sessionA new session from the same user, following an idle period
Cross-turnThe next turn within the same session
Cross-stepThe next model call within a single turn, following a tool result

The table below summarizes the reuse we measured for each mechanism on the models we run.

MechanismCross-userCross-sessionCross-turnCross-step
Explicit cache object, built per turnNoNoNoYes, including the first step, because the object is created before it
Implicit prefix cacheYes, for the shared head of the promptOn prefix equalityOn prefix equalityYes
Global automatic prefix cacheYesYesYesYes
Per-machine automatic prefix cacheNo; each user carries a distinct routing key and is served by a different nodeNo; the first request of a new conversation was uncached in every trialYesYes

This matrix is the central result of our investigation, and the remainder of this post follows from it. A mechanism that supports cross-turn and cross-user reuse rewards a prompt whose head never changes. A mechanism that supports only cross-step reuse is indifferent to it.

Locating the point of invalidation

Our investigation began with a discrepancy. Under the same prompt template and workload, one mechanism reported a far lower aggregate hit rate than another.

The aggregate figure proved misleading. When we broke the hit rate down by step index within a turn, every step after the first was almost fully cached, which is expected: each is the previous request extended by one tool result. The first step of each turn, however, was almost entirely uncached. This indicated that the prefix was being invalidated between turns, at a position near the head of the prompt.

To establish the order in which the prompt is serialized, we ran a single-variable probe against a synthetic prompt, changing one element at a time.

MutationEffect on the cached span
The tail of the user message, or appended historyNegligible
Replacing or appending a tool at the end of the schemaLarge loss
Replacing the first tool, or permuting tool orderLarger loss
Appending one line to the system promptComparable to permuting the tools
Prepending one line to the system promptTotal loss

These results imply a serialization order of system prompt, tool schema, then message history. A mutation in any one segment invalidates every segment that follows.

We then compared consecutive production requests byte by byte. The system prompt consisted of a static head followed by a dynamic tail, which contained retrieval results selected for the current query, per-user state, and the wall-clock time. This tail changed on every turn. We also found that the retrieval results were not deterministically ordered: an unchanged result set could serialize differently on two consecutive turns.

The point of invalidation was therefore inside the system prompt, ahead of both the tool schema and the entire message history. The tool schema was a second point of invalidation, since tools were selected per turn and both the set and its order varied.

Ordering segments by volatility

The layout we adopted follows one principle: segments are ordered by how frequently they change, from least to most volatile.

PositionSegmentVolatility
1Static instructionsConstant across users and turns
2The complete tool schema, sorted by nameConstant across users and turns
3Message historyAppend-only
4Per-turn context: retrieval results, per-user state, wall-clock timeChanges on every turn
5New user inputChanges on every turn

We evaluated this layout by replaying pairs of consecutive production turns, with the outputs scored blind by three independent judge models.

Freezing the tool schema in isolation provided no benefit. The dynamic context still preceded the schema, so the prefix was invalidated before the schema was reached, and the larger schema increased the cost of each turn.

Freezing the schema and relocating the dynamic context behind the history was effective. The first step of each turn moved from almost entirely uncached to mostly cached, the cost per turn fell substantially, and none of the three judges scored the outputs below those of the baseline layout.

Two of our findings were counterintuitive.

First, on the global cache, cold starts were cached as well as subsequent turns. Segments 1 and 2 are identical for all users, so the first request of a new session restores state that a different user's request created.

Second, a complete, frozen tool schema was less expensive than a smaller, dynamically selected one. A longer span that is cached costs less than a shorter span that is not, and dynamic tool selection is itself a source of invalidation.

Takeaway: the stability of the prefix matters more than its length. Content should be moved toward the tail of the prompt in proportion to how often it changes.

Requirements specific to each mechanism

The layout above is common to all mechanisms. Each mechanism then imposes one additional requirement.

MechanismOur approachFailure mode otherwise
Explicit cache objectWe create one object per turn with a short TTL, covering the full prefix including the most recent history; every step of the turn references itEvery step of the loop is billed as uncached input
Implicit prefix cacheWe do not use it; the explicit object on the same model performs betterNot applicable
Global automatic prefix cacheNo requirement beyond the layoutThe first step of each turn is uncached
Per-machine automatic prefix cacheWe attach a routing key that is stable per user to every request, derived from a hash of the user identifier and never the identifier itselfRequests are distributed across nodes, and the observed hit rate is zero

Per-machine caches

We should be upfront about an error in our own measurement. Our initial probe sent an identical prefix several times without any routing field and observed no cached tokens. We concluded that the model did not support prompt caching. That conclusion was incorrect: the cache existed, and our requests were being served by different nodes.

We then tested each routing field the provider documents, one at a time. One of them produced cache hits. The field recommended in the provider's own guide disabled caching.

Two further properties follow from node-local storage. Hits are best-effort, because a request will occasionally reach a node that has not yet built the entry. And we observed no cross-session reuse: with a byte-identical system prompt, the first request of every new conversation was uncached, regardless of how many earlier conversations had used the same routing key. The benefit of this mechanism is confined to the cross-turn and cross-step scopes.

Explicit cache objects

Because the object is rebuilt on every turn, a prefix that is stable across turns offers no advantage, and we left the layout for this model unchanged.

Rebuilding per turn can appear wasteful, but it has a property that is easy to overlook: the most recent history is always inside the object, so the entire prefix is billed at the cached rate from the first step onward. An object created once per session cannot incorporate history that is appended afterward. Whether per-turn creation is economical depends on how the provider bills for cache creation, and we recommend verifying this against invoices before relying on it.

Recommendations

Based on this work, we recommend the following practices for teams operating agents across more than one cache mechanism.

Measure the hit rate by step index. An aggregate figure can conceal a bimodal distribution in which one step is close to zero and the others are close to complete.

Compare consecutive production requests. The first differing byte identifies the point of invalidation. In our case it was the non-deterministic ordering of a retrieval result.

Make serialization deterministic. Sort the tool schema and any set-valued context. Unstable ordering invalidates the prefix even when the content has not changed.

Keep volatile content out of the head of the prompt. Timestamps, retrieval results and per-user state belong after the message history.

Avoid dynamic tool selection. Any change to the schema invalidates the history that follows it.

Probe routing fields before concluding that a model has no cache. Test one field at a time, and give more weight to the probe than to the documentation.

Treat layout as specific to the mechanism. The same change produced a large gain under one mechanism and none under another.

Verify against real requests before rollout. Inspect the usage fields for every step and confirm that the first step reports cached tokens, that the system prompt is byte-identical, and that the schema is the same sorted list.

Monitor continuously. We track the cached-token ratio, cost per turn and latency per turn for each model against a trailing baseline, and alert on a sharp decline. On one occasion a client library upgrade degraded the hit rate until it was corrected the following day.

Limitations and open questions

The gain we observed in production was smaller than the gain we measured in replay. The hit rate increased after rollout, but by less than the offline evaluation predicted, and we have not yet attributed the difference.

The underlying principles are well established: keep the prefix stable, keep the context append-only, and avoid changing tools in the middle of a task. What differs between providers is the scope of reuse that each mechanism offers. In our experience, that scope is not reliably described in documentation and has to be measured.

Result

With the layout and routing described in this post, we reduced the cost of our agent workload by roughly 50%, with no measurable loss in output quality.

References

  1. Anthropic. Prompt caching. Claude API documentation.
  2. Anthropic. Effective context engineering for AI agents.
  3. DeepSeek. Context Caching. DeepSeek API documentation.
  4. DeepSeek. DeepSeek API introduces Context Caching on Disk.
  5. Yichao "Peak" Ji. Context Engineering for AI Agents: Lessons from Building Manus.
  6. Cognition. Devin Fusion: Frontier Performance at 60% Lower Cost.
  7. Cognition. Introducing Fusion in Devin Desktop & CLI.
  8. Walden Yan. Don't Build Multi-Agents. Cognition.
  9. Walden Yan. Multi-Agents: What's Actually Working. Cognition.
  10. Cognition. Rebuilding Devin for Claude Sonnet 4.5: Lessons and Challenges.

← All posts