Hidden Factors That Drive Up LLM Costs
Model selection is the only LLM cost driver you can see on an invoice. Flagship models like Claude Opus cost more than economy models, such as Haiku. Invoices make the cost difference clear.
Hidden LLM cost drivers are the behavioral patterns in how agents are used that silently inflate token spend and drive up cost per successful task. Unlike model selection, which appears clearly on every invoice, these drivers are buried in the mechanics of LLM operations. They are the reasons why two teams using the same models can have dramatically different AI token usage patterns and monthly bills.
Retries are the largest hidden driver. Here is what hides them: a retry looks identical to a successful API call on the invoice. But a retry represents a task that failed to complete to satisfaction.
People have been trained to view retries as normal. So they just accept them. But, the tokens add up: the original prompt, the error output, the re-sent prompt, the new response. Nothing flags as waste because each line item is a valid API call. Helping the agent get the task right in fewer turns with more precise prompts, better context, and clearer explanations is the key to reducing daily token spend.
Cache utilization is the third driver. Every API call has the same format on the bill regardless of whether the model read from cache or recomputed from scratch. A session that preserves context across hours of is higly cost-effective because the savings compound turn after turn. They never appear as a line item you can point to.
Context bloat is the slowest accumulator. Redundant files, stale instructions, oversized context windows can spike your bills because they inflate every turn's token consumption across sessions.
Prompt overhead is the smallest and most persistent cost driver. Overhead can consist of irrelvant content formatting instructions, agent constraints restated in different ways, and re-explaining often-repeated tasks. Due to its dynamnic nature, much of this content is not cached. Overhead inflates prompts and increases llm costs.
The Five Hidden Cost Drivers Measured
This research, a behavioral analysis of 234,751 simulated LLM users, measured the impact of these variables on CPST:
- Retries: 5.00× CPST impact
- Cache: 3.16× CPST impact
- Context bloat: 1.59× CPST impact
- Prompt overhead: 1.42× CPST impact
Poor model selection can also inflate costs. However, in combination, these four hidden cost drivers vastly outweight model selection as a critical influncer of CPST.
How AI Agent Loops Multiply Costs
Retries are closely related to the issue of AI agent loops. This is when agents get stuck in repetitive behavior that burns tokens without progressing toward a goal. Understanding what an AI agent loop looks like and how to prevent it is one of the highest-leverage cost reduction strategies available.
How Cache Hits Affect AI Agent Token Usage
Cache behavior is another impactful cost driver. When the same information, such as a system prompt, tool definitions, or early conversation context, is sent back to the model it is not charged at full rates. The more content that remains static in LLM responses, the cheaper the call. This is the principle behind prompt caching in llms. Many LLM systems cache prompts automatically, but it is still important to understand how to implement it effectively (this is especially true in situations where the LLM API is being used directly).
Sessions that preserve the context cache across hours of can help lower CPST.
The Five Behavioral Variables That Drive AI Agent Costs
Here the five hidden behavioral variables that influence LLM costs are shown in additional detail.
The key insight is which factors should be lower or higher during operations.
Which Factors Should Be Lower
Three of the five variables should trend lower over time:
- Retry Rate: How many turns the agent takes versus the minimum needed to complete a task.
- Context Bloat: Redundant or stale tokens carried into each turn.
- Prompt Overhead: System prompts, tool schemas, and agent scaffolding as a fraction of the base task input.
- Premium Model Fraction: The fraction of requests using premium models that don't require frontier AI capabilities.
Which Factors Should Be Higher
Two of the five variables should trend higher:
- Cache Hit Rate: Share of input tokens served from the prompt cache instead of re-processed by the model on each turn.
Also worth mentioning is prompt optimization quality: Not directly shown in the chart, but implementing prompt optimization best practices such as providing agents with the right context, and clearly explaining desired outcomes, can reduce retry rates smf minimize context bloat and prompt overhead.
