Comparing Results with benchstat

Ten runs of one unchanged benchmark on one machine reported times from 1,419 to 1,735 ns/op, a spread of 22 percent. A second batch of ten runs, seconds later and with the same code, had a median of 1,682 ns/op against 1,475 for the first batch. That is a difference of 14 percent between a program and itself.

If unchanged code can differ from itself by 14 percent, then a change that appears to save 10 percent has proved nothing. How many runs does a comparison need, how should they be arranged, and what tells you that a difference is real? The tool that answers the last question is