Prefix Stability Across Four Prompt-Cache Mechanisms
·Codot Engineering
An agent runs as a loop of model calls. At each step the full context is sent again (instructions, tool schema, message history, tool results), and the model decodes a short structured output. The workload is therefore dominated by prefill: input tokens exceed output tokens by orders of magnitude. For such a system, both cost and time-to-first-token are determined largely by how much of that input the serving side can restore from a key-value (KV) cache instead of recomputing.
Prompt caching refers to the reuse of this precomputed state across requests. Reuse is prefix-based: a request can restore state only for the span of tokens that is identical to an earlier request from the first position, and the first differing token invalidates everything after it. Improving cache reuse therefore reduces to a single engineering problem: keeping the longest possible prefix byte-identical across the set of requests that are able to share it.
What is less often discussed is that this set of requests is not the same for every provider. We run several models in production, and across them we have encountered four distinct cache mechanisms, each with a different scope of reuse. In this post we classify those mechanisms, describe how we located the point at which our own prefix was being invalidated, and share the prompt layout and routing we now use for each.
Four cache mechanisms
Cache mechanisms can be distinguished along three dimensions: who creates a cache entry, where the entry is stored, and what a request must satisfy in order to hit it.
| Mechanism | Entry created by | Storage | Hit condition |
|---|---|---|---|
| Explicit cache object | The caller, through a separate call with a chosen time-to-live (TTL) | A named, addressable object | The request references the object before its TTL expires |
| Implicit prefix cache | The provider, at its discretion | Undocumented | Prefix equality with an earlier request, on a best-effort basis |
| Global automatic prefix cache | The provider, on every request | Shared, long-lived storage | Prefix equality with an earlier request from the first position |
| Per-machine automatic prefix cache | The provider, on every request | The memory of a single inference node | Prefix equality, and routing to the same node |
The first two are alternatives offered for the same model: the caller either constructs the cache explicitly or relies on the provider to cache what it can. We evaluated both on the same traffic. The implicit mode produced a substantially lower hit rate and a higher cost per turn, so we use the explicit mode.
Scopes of reuse
A cache entry is only valuable if a later request reuses it. For an agent, we find it useful to distinguish four nested scopes in which reuse can occur.
| Scope | Definition |
|---|---|
| Cross-user | The portion of the prompt that is identical for every user: instructions and tool schema |
| Cross-session | A new session from the same user, following an idle period |
| Cross-turn | The next turn within the same session |
| Cross-step | The next model call within a single turn, following a tool result |
The table below summarizes the reuse we measured for each mechanism on the models we run.
| Mechanism | Cross-user | Cross-session | Cross-turn | Cross-step |
|---|---|---|---|---|
| Explicit cache object, built per turn | No | No | No | Yes, including the first step, because the object is created before it |
| Implicit prefix cache | Yes, for the shared head of the prompt | On prefix equality | On prefix equality | Yes |
| Global automatic prefix cache | Yes | Yes | Yes | Yes |
| Per-machine automatic prefix cache | No; each user carries a distinct routing key and is served by a different node | No; the first request of a new conversation was uncached in every trial | Yes | Yes |
This matrix is the central result of our investigation, and the remainder of this post follows from it. A mechanism that supports cross-turn and cross-user reuse rewards a prompt whose head never changes. A mechanism that supports only cross-step reuse is indifferent to it.
Locating the point of invalidation
Our investigation began with a discrepancy. Under the same prompt template and workload, one mechanism reported a far lower aggregate hit rate than another.
The aggregate figure proved misleading. When we broke the hit rate down by step index within a turn, every step after the first was almost fully cached, which is expected: each is the previous request extended by one tool result. The first step of each turn, however, was almost entirely uncached. This indicated that the prefix was being invalidated between turns, at a position near the head of the prompt.
To establish the order in which the prompt is serialized, we ran a single-variable probe against a synthetic prompt, changing one element at a time.
| Mutation | Effect on the cached span |
|---|---|
| The tail of the user message, or appended history | Negligible |
| Replacing or appending a tool at the end of the schema | Large loss |
| Replacing the first tool, or permuting tool order | Larger loss |
| Appending one line to the system prompt | Comparable to permuting the tools |
| Prepending one line to the system prompt | Total loss |
These results imply a serialization order of system prompt, tool schema, then message history. A mutation in any one segment invalidates every segment that follows.
We then compared consecutive production requests byte by byte. The system prompt consisted of a static head followed by a dynamic tail, which contained retrieval results selected for the current query, per-user state, and the wall-clock time. This tail changed on every turn. We also found that the retrieval results were not deterministically ordered: an unchanged result set could serialize differently on two consecutive turns.
The point of invalidation was therefore inside the system prompt, ahead of both the tool schema and the entire message history. The tool schema was a second point of invalidation, since tools were selected per turn and both the set and its order varied.
Ordering segments by volatility
The layout we adopted follows one principle: segments are ordered by how frequently they change, from least to most volatile.
| Position | Segment | Volatility |
|---|---|---|
| 1 | Static instructions | Constant across users and turns |
| 2 | The complete tool schema, sorted by name | Constant across users and turns |
| 3 | Message history | Append-only |
| 4 | Per-turn context: retrieval results, per-user state, wall-clock time | Changes on every turn |
| 5 | New user input | Changes on every turn |
We evaluated this layout by replaying pairs of consecutive production turns, with the outputs scored blind by three independent judge models.
Freezing the tool schema in isolation provided no benefit. The dynamic context still preceded the schema, so the prefix was invalidated before the schema was reached, and the larger schema increased the cost of each turn.
Freezing the schema and relocating the dynamic context behind the history was effective. The first step of each turn moved from almost entirely uncached to mostly cached, the cost per turn fell substantially, and none of the three judges scored the outputs below those of the baseline layout.
Two of our findings were counterintuitive.
First, on the global cache, cold starts were cached as well as subsequent turns. Segments 1 and 2 are identical for all users, so the first request of a new session restores state that a different user's request created.
Second, a complete, frozen tool schema was less expensive than a smaller, dynamically selected one. A longer span that is cached costs less than a shorter span that is not, and dynamic tool selection is itself a source of invalidation.
Takeaway: the stability of the prefix matters more than its length. Content should be moved toward the tail of the prompt in proportion to how often it changes.
Requirements specific to each mechanism
The layout above is common to all mechanisms. Each mechanism then imposes one additional requirement.
| Mechanism | Our approach | Failure mode otherwise |
|---|---|---|
| Explicit cache object | We create one object per turn with a short TTL, covering the full prefix including the most recent history; every step of the turn references it | Every step of the loop is billed as uncached input |
| Implicit prefix cache | We do not use it; the explicit object on the same model performs better | Not applicable |
| Global automatic prefix cache | No requirement beyond the layout | The first step of each turn is uncached |
| Per-machine automatic prefix cache | We attach a routing key that is stable per user to every request, derived from a hash of the user identifier and never the identifier itself | Requests are distributed across nodes, and the observed hit rate is zero |
Per-machine caches
We should be upfront about an error in our own measurement. Our initial probe sent an identical prefix several times without any routing field and observed no cached tokens. We concluded that the model did not support prompt caching. That conclusion was incorrect: the cache existed, and our requests were being served by different nodes.
We then tested each routing field the provider documents, one at a time. One of them produced cache hits. The field recommended in the provider's own guide disabled caching.
Two further properties follow from node-local storage. Hits are best-effort, because a request will occasionally reach a node that has not yet built the entry. And we observed no cross-session reuse: with a byte-identical system prompt, the first request of every new conversation was uncached, regardless of how many earlier conversations had used the same routing key. The benefit of this mechanism is confined to the cross-turn and cross-step scopes.
Explicit cache objects
Because the object is rebuilt on every turn, a prefix that is stable across turns offers no advantage, and we left the layout for this model unchanged.
Rebuilding per turn can appear wasteful, but it has a property that is easy to overlook: the most recent history is always inside the object, so the entire prefix is billed at the cached rate from the first step onward. An object created once per session cannot incorporate history that is appended afterward. Whether per-turn creation is economical depends on how the provider bills for cache creation, and we recommend verifying this against invoices before relying on it.
Recommendations
Based on this work, we recommend the following practices for teams operating agents across more than one cache mechanism.
Measure the hit rate by step index. An aggregate figure can conceal a bimodal distribution in which one step is close to zero and the others are close to complete.
Compare consecutive production requests. The first differing byte identifies the point of invalidation. In our case it was the non-deterministic ordering of a retrieval result.
Make serialization deterministic. Sort the tool schema and any set-valued context. Unstable ordering invalidates the prefix even when the content has not changed.
Keep volatile content out of the head of the prompt. Timestamps, retrieval results and per-user state belong after the message history.
Avoid dynamic tool selection. Any change to the schema invalidates the history that follows it.
Probe routing fields before concluding that a model has no cache. Test one field at a time, and give more weight to the probe than to the documentation.
Treat layout as specific to the mechanism. The same change produced a large gain under one mechanism and none under another.
Verify against real requests before rollout. Inspect the usage fields for every step and confirm that the first step reports cached tokens, that the system prompt is byte-identical, and that the schema is the same sorted list.
Monitor continuously. We track the cached-token ratio, cost per turn and latency per turn for each model against a trailing baseline, and alert on a sharp decline. On one occasion a client library upgrade degraded the hit rate until it was corrected the following day.
Limitations and open questions
The gain we observed in production was smaller than the gain we measured in replay. The hit rate increased after rollout, but by less than the offline evaluation predicted, and we have not yet attributed the difference.
The underlying principles are well established: keep the prefix stable, keep the context append-only, and avoid changing tools in the middle of a task. What differs between providers is the scope of reuse that each mechanism offers. In our experience, that scope is not reliably described in documentation and has to be measured.
Result
With the layout and routing described in this post, we reduced the cost of our agent workload by roughly 50%, with no measurable loss in output quality.
References
- Anthropic. Prompt caching. Claude API documentation.
- Anthropic. Effective context engineering for AI agents.
- DeepSeek. Context Caching. DeepSeek API documentation.
- DeepSeek. DeepSeek API introduces Context Caching on Disk.
- Yichao "Peak" Ji. Context Engineering for AI Agents: Lessons from Building Manus.
- Cognition. Devin Fusion: Frontier Performance at 60% Lower Cost.
- Cognition. Introducing Fusion in Devin Desktop & CLI.
- Walden Yan. Don't Build Multi-Agents. Cognition.
- Walden Yan. Multi-Agents: What's Actually Working. Cognition.
- Cognition. Rebuilding Devin for Claude Sonnet 4.5: Lessons and Challenges.