The METR AI study: devs said 20% faster, the clock said 19% slower

Sixteen experienced open-source developers worked through 246 real tasks on their own repositories while a randomized trial timed them. When they were allowed to use AI tools, they finished 19% slower. Afterwards, the same developers estimated the AI had made them 20% faster. That is the headline result of the METR AI study, and a year later it is still the most uncomfortable data point in AI productivity measurement. Not because the tools were slow. Because the humans were certain, and the humans were wrong by 39 points, in the wrong direction.

What the METR AI study actually measured

The setup deserves respect, because almost nobody measures this properly. METR recruited experienced maintainers, averaging five years of work on their own repositories, most of them over a million lines of code. Each real issue was randomly assigned: AI allowed or AI forbidden. The AI condition mostly meant Cursor Pro with Claude 3.5 and 3.7 Sonnet, the frontier tooling of early 2025. Screen recordings and wall-clock time settled what happened, and the full paper is on arXiv for anyone who wants to check the method.

Every layer of human judgment in that experiment pointed the same way, and every layer missed. Economics experts forecast a 39% speedup. Machine learning experts forecast 38%. The developers themselves predicted 24% before starting, and after finishing they still reported a 20% gain. The stopwatch recorded a 19% slowdown.

One study, early-2025 tools, experienced developers on mature codebases they knew deeply: the authors are careful about scope, and so should you be. METR itself announced a redesigned follow-up experiment in February 2026 to measure uplift with more precision. But the scoped claim is already the important one. Under the most favorable measurement conditions available, perceived time saved and measured time saved did not just diverge. They flipped sign.

Self-reported time savings are not noisy, they are directionally wrong

Noise would be fine. If self-reports scattered around the true value, you could survey enough people and average your way to the truth. A 39-point miss with the sign flipped is not scatter. It is bias, and it points the way every budget wants it to point.

Now look at how the industry produces its numbers. Atlassian reports 2 to 3 hours saved per developer per week in its AI-native SDLC push. Vendor case studies, adoption surveys, and internal AI dashboards overwhelmingly run on the same fuel: someone asked the users how much time the AI saved them, and the users answered. The METR result says that answer can be wrong not by 10 or 20 percent, but by the full width of the claim, from experienced professionals with every incentive to know their own workflow.

If the people best positioned in the world to estimate their own time savings got the direction wrong, a survey of your own team has no floor under it. "Our engineers report saving four hours a week" is not a measurement. It is a mood.

Why perception fails here

The mechanism is mundane. Time spent prompting, waiting for generations, and reviewing output does not register as cost, because it feels like the work is happening. The developer is active, the screen is moving, progress is visible. Meanwhile the counterfactual, how long the task would have taken solo, is never observed, only imagined. Humans fill unobserved counterfactuals with optimism.

This also explains why the bias survives contact with the result. METR's developers reported the 20% speedup after completing the tasks, with the actual duration behind them. Experience did not correct the estimate, because there was no measured baseline in view to correct against.

Put a measured baseline behind the dashboard

The fix is not to distrust AI. The AI tooling of 2026 is not the AI tooling of early 2025, and plenty of agent deployments produce real, large savings. The fix is to stop letting perception be the measurement instrument. Two rules follow directly from the METR result.

First, the human baseline must come from observation, not recall. Before an agent takes over a task, time a person doing it. A stopwatch and five samples beat any estimate a workshop will produce. Record where the number came from, because "says who" is the first question in every review.

Second, savings must be logged per run against that baseline, not reconstructed per quarter from memory. One HTTP call at the end of each successful run is enough:

curl -X POST https://humanhours.dev/api/v1/track \
  -H "Authorization: Bearer hh_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "agent_id": "intake-triage",
    "task_type": "ticket_classification",
    "outcome": "success",
    "human_baseline_minutes": 9,
    "metadata": { "baseline_source": "stopwatch, 5 samples, 2026-07" }
  }'

At typical production volumes, somewhere between 1,000 and 100,000 agent runs a month, this produces an hours-saved and money-saved number with a timestamped trail behind every unit. When the number is challenged, and after METR it should be challenged, the answer is a measured baseline and a run log, not a survey.

Trust the stopwatch

The METR AI study did not prove AI slows everyone down. It proved something more useful: the gap between felt productivity and measured productivity is large enough to invert the conclusion. Every ROI dashboard built on self-reported time savings inherits that gap silently.

So keep the dashboard. Change its foundation. Measured baselines, per-run records, wall-clock arithmetic. If your AI savings are real, measurement makes them undeniable. If they are not, better to be the one holding the stopwatch than the one holding the survey.