What is the easiest straightforward way of telling which version performs better?

Question

I have an application, which I have re-factored so that I believe it is now faster. One can't possibly feel the difference, but in theory, the application should run faster. Normally I would not care, but as this is part of my project for my master's degree, I would like to support my claim that the re-factoring did not only lead to improved design and 'higher quality', but also an increase in performance of the application (a small toy-thing - a train set simulation).

I have toyed with the latest VisualVM thing today for about four hours but I couldn't get anything helpful out of it. There isn't (or I haven't found it) a way to simply compare the profiling results taken from the two versions (pre- and post- refactoring).

What would be the easiest, the most straightforward way of simply telling the slower from the faster version of the application. The difference of the two must have had an impact on the performance. Thank you.

Answer 1

I would suggest creating a few automated tests that simulate real usage of the application. Create enough tests to have a decent benchmark.

Run the test suite for both versions of the app under various loads.

That should give you a pretty real world time of performance. Doing it at a lower level may not give you an accurate truth.

Answer 2

I assume you can find a good way to measure the difference, and you can say it is due to the refactoring if there's nothing else you did, but I would be bothered by that, because that's not really understanding why it's faster.

Here's an example of really aggressive performance tuning.

What convinces me is if it can be shown that

1. A particular line of code, or small set of lines, is directly responsible for approximate fraction F% of overall wall-clock time,

2. It is shown that that line or lines is not really necessary in the sense that a way can be found to use it a lot less or maybe not at all,

3. That change results in a reduction in overall wall-clock time of approximately F%.