Proactive Prompt Caching Can Deliver Significant LLM Cost Savings
Caching calls to LLMs can be a high impact cost-savings strategy. But, many may not understand it well. Caching is when data sent to an LLM is stored for later use. When the same information, such as a system prompt, is sent back to the model, it is not charged at full rates. The more content that remains static in LLM responses, the cheaper the call.
Prompt Caching for LLMs Explained
Prompt caching for AI agents works by storing conversation prefixes, which includes the system prompt, tool definitions, and early conversation context. Upon repeated requests, this cached data enables the use of previously computed data instead of reprocessing it from scratch.
Note: Adding content to the cache in some cases can be more expensive than a regular model API call because the data has to be stored. However, the cost-savings compound as the model receives static information repeatedly over time.
Major providers handle caching differently. For example, OpenAI caches automatically for prompts of 1,024 tokens or longer, with no manual configuration required. Cache routing uses a hash of the initial prefix to route requests to servers that recently processed the same content, achieving up to 90% input token cost reduction on cache hits.
Anthropic requires explicit cache_control breakpoints at the block level, giving fine-grained control over what gets cached. Cache reads cost 10% of base input tokens, while 5-minute cache writes cost 25% more.
Amazon Bedrock uses cache checkpoints defined in the cachePoint parameter, with per-model token minimums (ranging from 1,024 to 4,096 tokens depending on the model) and a maximum of 4 checkpoints per request.
All three providers share the same fundamental principle: place stable, reusable content at the beginning of the prompt and variable content at the end.
How Prompt Caching Reduces AI Agent Costs
The simulation results indicate that prompt cache hit rates have a linear relationship with cost. At a 25% cache hit rate, monthly cost per successful tasks average $1,046. At 85% cache hit rate, costs drop to $408. The savings observed during the simulation from improving cache behavior are substantial: $638 per month. Savings accumulate steadily with every point of improvement. Every session that misses the cache re-pays the full input bill.
Cache Savings Scale with Model Cost
Improving prompt cache hit rates save more money when expensive models are used. Simulated flagship LLM users moving from 25% to 85% cache rates saved roughly $933 per month. Standard tier saves $657. Economy tier saves $371. The same cache behavior improvement yields different dollar amounts depending on model used, but the principle is the same: higher cache hit rates reduce costs across all tiers.
Prompt caching strategies differ in implementation, but share the same economic principle: every byte that lands in the cache saves the full compute cost of reprocessing it.
When to Clear the Cache
Optimizing for the cache is a critical strategy, but goes up against a piece of conventional wisdom: keep the LLM context lean because the more information that an LLM has to process (old responses, data, etc.) the less it can perform optimally. This phenomon is commonly referred to as context rot.
However, every time a model has to start from scratch (after the conversation context has been removed), the less information is in the cache, resulting in higher costs.
So what should users do? Save all conversations or take the cost hit for higher-quality LLM interactions with small context loads?
It all comes down to having a good strategy.
Clear the context window when a new task is being started. Old information is not going to help during an unrelated conversation.
If a conversation is going well: compact. This will prevent context rot and create a summary of the conversation, and oftentimes key details can be retained by adding them to the compacted conversation context. Compacting clears the cache, but still has cost-saving benefits due to the smaller context window size. Only compact proactively if LLM response quality is degrading significantly.
The Context Contamination Problem
There is another time when clearing a conversation and starting over makes a lot of sense: when the context is contaminated. In a May 2026 study, Zhanfu Yang, described one aspect of this phenomenon as: "When an LLM agent fails a multi-step tool-augmented task and retries, the failed attempt typically remains in its context window—contaminating the next attempt and elevating the per-step error rate beyond the base level."
What this means in practice is that bad data regarding failed task attempts makes it more likely an LLM will fail again. This causes retries and elevates costs due to failed task attempts. When context is clearly contaminated, that is a sign that it is better to clear the conversation (and the cache) and start over. The cost of retries and failed tasks outweighs the benefits of caching.
Why Cache Optimization Cannot Fully Limit AI Agent Costs Alone
High cache hit rates alone does not maximally lower monthly costs. If other behaviors are not optimized, cost savings due to cache improvements can be eliminated due to other LLM use inefficiencies.
The Eager Conversationalist has the highest cache hit rate at 74% — above the Whisperer's 66% — but still pays $298 per month more than baseline. The Free Spender has a lower cache rate of 56% but pays $866 per month more. The Stubborn Economizer and Precise Spender have the lowest cache rates at 38% and 40%, yet their cost penalties of $452 and $544 per month come primarily from other behaviors.
Cache optimization reduces costs, but it cannot compensate for poor choices elsewhere. A higher cache rate on its own is not a silver bullet cost solution. The cost drivers that dominate are model selection, retry behavior, and context bloat, factors that cache optimization cannot fully compensate for.
Implementing AI Prompt Optimization Best Practices
Over time, effective AI prompt optimization strategies have aligned around key best practices such as:
- Maximizing cache hits over hours of agent operations by carefully ordering prompts (including putting static content first in the prompt)
- Understnding whether prompt caching is done automatically or requires manual setup. Specifically, many command line (CLIs), Web intefaces and some APIs manage prompt cache management automatically for users. Despite this, taking actions such as switching models, compacting the conversation and changing model effort levels will clear the cache
For users utilizing APIs, the prompt cache management approach will depend on the agent framework. Generally, the best appraoch is to pay attention to the order of prompt context. For example,Anthropic recommends putting "static ontent first, dynamic content last", e.g,:
- First: Static content such as the system prompt, tool defintions
- Second: AGENT.MD, rules files other rarely changing instructions
- Third: Session context related to the task being worked on
- Fourth: Conversation messages, which in general aren't cached because they change often
This strategy will help reduce cache misses, or when information sent to the LLM isn't detected as being part of the cache. This is because the majority of the prompt sent to the llm will consist of static information.
Semantic Caching: An Important Strategy in Agentic Worflows
Semantic caching is an optimization technique for LLM applications that retrieves cached responses based on the semantic meaning of a query rather than an exact text match. This is a high-ROI caching strategy because matchihng text by meaning rather than keywords can greatly increase prompt cache hit rates.
Traditional caching often requires an exact keyword match such as "Champion's League 2024 winner." In semantic caching, the phrase: Champion's League final result" is recognized as having the same meaning and intent, and a similar answer can be returned by the LLM.
Semantic caching is importnt for AI agents because they often combine LLM calls for a variety of tasks, such as planning, calling tools and verifying results. Using semantic caching, response times and costs can be reduced.
