Inconsistent Java performance

I have an interpreter written in Java I tried to test the performance results of various optimizations in the interpreter To do this, I parse the code and run the interpreter repeatedly on the code until I get five runs. The difference between these runs is very small (the following time is 0.1 seconds). Take the average value and print it There is no I / O or randomness in the interpreter If I run the interpreter again, I will get a different run time:

91.8s
95.7s
93.8s
97.6s
94.6s
94.6s
107.4s

I tried not to use server and client VMS, serial and parallel GC, large tables and windows, and Linux These are in 1.6 0_ 14. On the JVM The computer has no processes running in the background So I asked what might have caused these big changes, or how could I find out what it was?

Practical problems arise because the program must iterate to a fixed point solution and the values are stored in a hash set Different hash values between runs lead to different ordering, which in turn leads to changes in the number of iterations required to reach the solution

Solution

"Wall clock time" is rarely a good measure of benchmarking Modern operating systems are highly unlikely to "[no processes running in the background" – it is well known that it may write dirty block buffers to disk because it determines that there is no other contention

Instead, I recommend using threadmxbean to track actual CPU consumption