A field guide to five different mechanisms

Caching in LLM agent systems

“The cache” is usually several unrelated things. One lives only inside a generation. Another reuses an exact prompt beginning across requests. Others belong to the agent application, its tools, or its context manager.

Prompted by Luke Parker’s post and a question about long-running orchestrators
The short answer

A dormant agent thread does not spend tokens, and its existence does not keep a provider prompt cache warm.

When the harness wakes and makes another model request, that request may reuse an eligible exact prefix—or it may be cold because the prefix changed, the entry expired, or routing did not find it. Context saved by the harness is durable application state; a provider cache is temporary inference infrastructure.

One request, several reuse boundaries

Follow the solid line for model work. The dashed line is a tool round trip. Each cache matches a different thing and has a different owner.

application provider current generation
Architecture of caching in a tool-using LLM agent A user request moves through an agent harness, an application semantic cache, a provider exact-prefix cache, model prefill, and a token-by-token decode loop. Tool calls branch through application data caches and return to the model. Labels state what each layer matches, who owns it, and what invalidates it. APPLICATION / HARNESS MODEL PROVIDER APPLICATION / INFRASTRUCTURE User new intent or event Agent harness OWNS: prompt + context history · tool schemas/results compaction · checkpoints child agents · wait/wake Semantic decision MATCH: similar meaning embedding + threshold metadata · auth · freshness bad boundary → bypass New result answer + new cacheable data safe hit Exact-prefix cache MATCH: exact beginning eligibility + route/key + TTL stores processed prefix KV breaks at early edit / expiry Model prefill PROCESS: input read warm prefix KV compute uncached suffix write eligible prefix Decode loop MATCH: earlier tokens here retain keys/values generate next token ends with this generation miss / bypass → model request Tool & data caches HTTP · search · file index · embeddings · builds · tool outputs tool call tool result Mobile architecture of caching in a tool-using LLM agent A vertical flow shows the user, agent harness, semantic cache decision, provider exact-prefix cache, model prefill, token decode loop, tool and data caches, and the new result. Each cache states its owner, match rule, and invalidation boundary. APPLICATION / HARNESS User or event starts a request Agent harness / prompt builder OWNS: history + tool context compaction · checkpoints · child agents · wait/wake Semantic-cache decision MATCH: meaning + filters + freshness app-owned · wrong boundary means bypass MODEL PROVIDER Exact-prefix cache MATCH: exact beginning + route/key + TTL stores processed prefix KV · early edit/expiry breaks it Model prefill READ warm prefix · COMPUTE cold suffix may write an eligible prefix for later requests Per-generation KV decode loop MATCH: earlier tokens in this response retain keys/values → generate next token lives only for the active generation model request on miss/bypass APPLICATION / INFRASTRUCTURE Tool & data caches MATCH: URL · query · file hash · version HTTP · search · indexes · embeddings · builds · outputs tool call result New result answer + cacheable application data
The same key/value representation underlies both model-side layers, but the lifetime and reuse boundary differ: decode KV avoids recomputation inside one generation; prompt caching tries to reuse prefill work across separate requests.

Warm, cold, and “similar” are different states

Same exact beginning Eligible prefix, compatible routing/key, still retained PROVIDER HIT
Same beginning, later Entry expired or the request did not reach matching cache state PROVIDER MISS
Changed beginning Early instructions, tool schemas, ordering, image settings, or metadata changed MISS AFTER DIVERGENCE
Different words, same intent Not an exact-prefix match; an application embedding may judge it close enough SEMANTIC MAYBE
Appending a new user turn usually preserves the earlier prefix. Editing a system instruction near the front invalidates that point and everything after it. A semantic hit is a separate application decision, not a looser provider prompt-cache match.

Five layers, five contracts

Layer Match & stored form Lifetime & owner Benefit & principal failure
In-generation KV Key / match Earlier tokens in this active decode Stored Attention keys and values by layer Lifetime Active generation / serving policy Owner Model runtime Benefit Avoid recomputing all prior-token KV for every next token Failure Memory pressure; ends with or is evicted from the generation
Provider prompt / prefix Key / match Exact eligible beginning, often plus routing/cache key Stored Processed prefix KV from prefill Lifetime Provider/model policy and retention window Owner Model provider Benefit Lower repeated prefill latency and often cost Failure Early change, threshold, route, expiry, or eviction causes a cold miss
Semantic cache Key / match Embedding similarity above a chosen threshold, plus filters Stored Query vector and prior answer or tool result Lifetime App TTL, freshness, model/index version Owner Application or gateway Benefit Reuse across different wording Failure Plausible but wrong, stale, personalized, or unauthorized response
Tool & data caches Key / match URL, request, search query, file hash, commit, package, or tool inputs Stored HTTP payload, index, embedding result, build artifact, tool output Lifetime Source-specific TTL/version/eviction Owner Harness, tool, proxy, CI, or data layer Benefit Avoid repeated I/O and deterministic work Failure Stale data, incomplete key, tenant leak, or bad invalidation
Harness context Key / match Conversation/session/checkpoint identity Stored Messages, summaries, tool schemas/results, files, task state Lifetime Session or durable application storage Owner Agent harness Benefit Continuity; compaction can reduce input size Failure Lost detail, stale state, prefix reshaping, or child-agent cold start

Worked example: a PR-review orchestrator wakes up

The task is “Review PR #184 against our repository rules and report failing checks.” The same job passes through every layer, but each layer reuses something different.

1 · Harness
Builds the request from stable review instructions, tool schemas, a compact task checkpoint, and the new wake event. It may omit old chatter to reduce context. That saves input tokens, but reshaping the beginning can reduce prefix reuse.
2 · Semantic gate
May reuse a prior explanation such as “what does this lint rule mean?” only if the embedding is close enough and metadata agrees: repository, rules version, audience, authorization, and freshness. It should not reuse “PR #183 is green” for PR #184.
3 · Provider prefix
Can hit on stable system instructions and unchanged tool definitions. Appending PR #184 late in the request preserves that beginning. Injecting a timestamp or changing a tool schema near the front creates a new prefix from that point.
4 · Prefill + decode
The provider reads any warm prefix KV, computes the uncached suffix, then generates a fresh answer. During generation, the decode KV retains attention keys/values for prior tokens so the model does not rebuild them for every next token.
5 · Tools
GitHub metadata might be cached by repository + PR + head SHA; CI results by workflow run; a file index by commit; dependencies by lockfile hash. Those are harness/data caches. The head SHA belongs in the key so new code cannot inherit old proof.
6 · Wait
The harness stores a checkpoint and subscribes to the next CI event. While it is waiting, no model request is happening, so the dormant thread itself consumes no tokens. On wake, the provider prefix may still be warm—or may need to be rebuilt.

The useful design target is not “never go cold.” Keep stable prompt material early, volatile facts late, tool cache keys complete, checkpoints small but sufficient, and waits event-driven. Then measure actual cache reads/writes instead of treating a long-lived thread as proof of a warm cache.

Provider behavior is a dated implementation detail

The durable model is “exact prefix + eligibility + routing + retention.” Thresholds, TTLs, storage modes, and prices can change by provider, API, model, account policy, and date.

OpenAI

Checked 23 Jul 2026

Current API docs describe exact-prefix hits and a 1,024-token eligibility floor. For GPT-5.6 and later families, cache writes are billed at 1.25× uncached input and the documented default/minimum TTL is 30 minutes; the service may retain entries longer. Older-model in-memory and extended-retention policies differ.

Anthropic

Checked 23 Jul 2026

Current Claude API docs require a 100% identical cached prefix. The default ephemeral TTL is 5 minutes and refreshes on a hit; a 1-hour option costs more. Published multipliers are 1.25× base input for 5-minute writes, 2× for 1-hour writes, and 0.1× for reads.

Google

Docs updated 7 Jul 2026

Gemini Interactions API docs say implicit caching is enabled for Gemini 2.5 and newer, with current minimums of 2,048 or 4,096 input tokens depending on model. Google recommends common content at the beginning and nearby requests. Pricing is model-specific; explicit cache products can also charge for retained-token storage.

Evidence boundary & further reading

This is an architecture guide, not a benchmark. It uses no invented hit rates, cost savings, or latency claims. Provider facts above are scoped to their public API documentation as checked on 23 July 2026; verify the linked pages before relying on them for billing, privacy, or retention decisions.