Study: Don't Put Your Brain on a Shelf: Expertise Matters in LLM Use

On June 16, 2026, Anthropic published a study of 400,000 Claude Code sessions (featuring more than 230,000 users) conducted between October 2025 and April 2026. The study is a continuation of Anthropic's recent work focusing on how LLMs are changing the nature of work.

The study comes at a time when many are questioning whether the use of AI tools is leading to skills decline, a kind of cognitive deficit that comes from outsourcing key tasks to LLMs.

One of the surprising results of the study was that a user's domain expertise had a significant impact on their ability to use LLMs successfully. Domain expertise was the primary driver of task success.

Another interesting finding was that users did not have to be experts to get good results from LLMs. Task success increased rapidly until users were at an intermediate proficiency, and leveled off at the advanced and expert levels.

Anthropic also observed that domain expertise scales what AI coding agents can produce per prompt:

  • Expert users generated 2.4x more actions and 5.3x more output words per prompt than novices.
  • Each expertise level increase produced approximately 9% more actions and 13% more output per prompt.

One question that arises from this analysis is whether the additional LLM outputs result in economically beneficial work. Anthropic observed that the economic value of LLM-aided tasks increased over the study period. However, cost pressures, and questions about how to use LLMs efficiently, and deliver ROI persist.

This is because organizations and individuals are spending large sums on inference. For example, Microsoft ordered its engineers to stop using Claude Code because costs per engineer per month was about $2,000. Microsoft was able to move its engineers to its own solution, Copilot which charges a flat fee for inference However, many AI labs are moving from flat-rate pricing to credit-based frameworks, which are more expensive.

In an environment where access to inference is becoming increasingly expensive, it's critical for individuals, teams and organizations to have good information about:

  • How to maximize time spent working with LLMs
  • What are the habits and behaviors that contribute to poor outcomes when using LLMs?
  • What are the financial costs and benefits of poor LLM use habits?

To find out, we developed a study examining how 234,751 simulated LLM users.