Is the difference real, or just noise?
You tried a new way of working and it felt faster. Whether five timings can prove it depends less on the timings than on how you collected them. Here are the same ten numbers, read two ways.
The same numbers, two designs
A took 40, 55, 30, 70, and 45 minutes. B took 35, 46, 26, 60, and 38. If those are ten different tasks, the gap between a 26-minute task and a 70-minute one swamps the 7 minutes B saves. If they’re the same five tasks done both ways, each task is compared with itself, and that spread drops out.
Can’t tell. The data is consistent with anything from B being 13.6 minutes slower to 27.6 minutes faster per task.
A − B: 7.0 minutes, 95% interval [−13.6, 27.6].
You’d need about 64 tasks per condition to reliably see a difference this size (7.0 minutes).
One dot per timing. The bar under each row is its mean, with a 95% interval. Without pairing, every task’s own size is noise, and the two rows overlap.
Try your own numbers
Minutes per task, one row per task. Blank cells are skipped. Nothing you type leaves this page.
| Task | A minutes | B minutes |
|---|---|---|
| 1 | ||
| 2 | ||
| 3 | ||
| 4 | ||
| 5 |
Fill in both times for at least two tasks to get a verdict.
Feeling faster isn’t a measurement
In METR’s randomized trial, published in July 2025, 16 experienced open-source maintainers worked 246 real issues. Before starting, they forecast that AI tools would speed them up by 24%. Measured, they took 19% longer. Afterward, they still estimated they’d been 20% faster.
METR has since revised its numbers and changed its study design (February 2026): developers had started refusing to work without AI, which skewed who and what got measured, and METR now calls its newer data only very weak evidence either way. So don’t read this as “AI makes you slower.” The lesson that survives is narrower: the perception error kept its sign even after the work was done.
- METR forecast before the work: +24% (faster).
- METR measured: −19% (19% longer per issue).
- METR participants’ estimate afterward: +20% (faster).
What to measure, and how this works
Measure the time to an accepted result, not to a first draft. Count rework (how often someone has to touch it again), review minutes, cost per accepted result rather than per run, and the bugs that got past review. A way of working that’s 20% faster but doubles rework hasn’t saved you anything, so pick the one endpoint you care about before you start, and measure a baseline before you change anything.
Don’t measure lines of code, suggestion acceptance rates (they go up when you stop reading), raw token counts, or how many agents you have running. None of them is an outcome, so no amount of data about them can tell you whether the work got better. “We didn’t measure it” is a more honest answer than any of those.
The verdict is a 95% t interval for A − B. Unpaired, it’s Welch’s interval, which doesn’t assume the two groups vary the same amount. Paired, it’s a one-sample interval on each task’s own difference. If the interval leaves out zero, the data can tell A and B apart. The task-count sentence uses the normal approximation for a two-sided test at α = 0.05 with 80% power, taking the spread and the difference from your data.