Expertise Matters in LLM Use, Up to a Point

Anthropic's analysis of ~400,000 Claude Code sessions revealed a relationship between domain expertise and task success. Novices had a verified success rate of 14.5%. Intermediate and above users achieved verified success rates of 28–33%. Anthropic defined verified success as code pushed to Github, verbal confirmation of success and other concrete completion actions.

More experienced users get the agent to do more per prompt. Agents take nine percent more actions and deliver thirteen percent more outputs per expertise level. Specifically, an expert prompt sets off roughly twelve actions and 3,200 words of output. A novice prompt sets off about five actions and 600 words.

How Expertise Affects Cost Per Successful Task

Expertise improves task outcomes. But can it also boost cost efficiency?

In the simulation CPST was measured across five expertise levels defined by Anthroopic (Novice, Beginner, Intermediate, Advanced and Expert) within each behavioral archetype. The pattern is consistent but the magnitude varies. For the LLM Whisperer, CPST declined by 30 percent at the intermediate level: from about $10 for novices down to $7 for simulated intermediate LLM users. Beyond this point, cost saving gains flatten. Advanced and expert Whisperers average close to $8 respectively. The same flattening pattern holds across all five behavioral profiles.

Why Intermediate Users Get the Most Value

Domain expertise improves task completion, which delivers financial value up to the intermediate level. But deep specialization yields only a modest return on CPST. The expertise-cost flattening pattern means that investing in beginner to intermediate-level AI agent skill training may deliver the majority of benefits, with diminishing returns at advanced and expert levels.

This finding challenges the assumption that more expertise always equals better economics. The truth is more nuanced. Behavioral discipline at the intermediate level may provide more cost efficiency gains than expertise alone.