Performance Testing

Question

Hey all, We are developing automated regression tests using VMWare and NUnit. We have divided tests into steps and now I would like to see each step be examined for performance regression. Simply timing the tests, as NUnit does, does not seem reliable. I have figured in a acceptance factor of about 15% but our steps can differ sometimes to over 35%. In such a resource dependent test environment is there any consistent way of testing performance? Is a "smart" timing system my only option?

Answer 1

You might look into the features available with a tool such as Ants Profiler as it does give method executing/run times, but I'm not sure what it offers in terms of repeated testing.

Answer 2

With respect to performance testing I've been very skeptical of using vmware or other virtualization processes. The way we have handled this in the past is to have part of the build install the latest version on a static machine and run the tests. You should see more consistent results outside of the virtualization.

Answer 3

For this sort of performance testing, there's no such thing as a system that will give you a simple pass/fail result. In real life, changing your system is likely to make some things faster and some other things slower, so it's usually not a choice between "better" and "not better", it's a choice between different kinds of better. (Of course, you want to avoid cases where it's strictly worse.)

What I've done for this in the past is to just keep statistics over time. Every time you run your tests, drop the results in a SQL database with the revision number and the test timings. Then you can graph them whenever and however you want (ideally in a little web applet so everyone on the team can review them) and see if your performance is trending up or down, or if performance has been sucking ever since a particular revision.

The key thing here, though, is that it needs to be a graph. That way human eyes can look at it and find the trends. You could spend all week trying to come up with an AI algorithm to analyse the data numerically, but it would never beat a human's pattern-recognition ability.