what is the best tool for performance regression testing

Question

our organization is looking for a tool to help with performance testing on each release. We ship a whole bunch of new software and we want to ensure that performance on key functions has not slowed down since the last prod release. We have code in C# and Java. This can be anything from:

1. when i run this function it takes < 2 seconds
2. when i run this set of functions the total < 5 seconds
3. etc . .

Is it best to do this using our basic unit testing continuous integration (nunit, junit, team city) with hand written performance checks or are there specific tools that can help focus on on this area.

Any suggestions?

Answer 1

What we use is python to write scripts + extensive logging to generate XML logs which can be then imported into spreadsheet.

Answer 2

I have been tinkering with a tool called 'Basher' that allows you to write "tasks" that are picked up by the system and subsequently run for a configurable amount of time (to allow the JVM to warm up for example) and then performs a run, recording task execution time, averages and the like.

The 1.0 version has been quietly around for a while and if you care to take a look, it is available at http://basher.sourceforge.net

The trunk version contains some improvements - there is maven integration, with configurations being specified in the pom.xml, the bare bones of a reporting framework, etc.

Answer 3

On my projects (which tend to use Spring), I use the AOP and the PerformanceMonitorInterceptor.

While you may not use Spring, it's definitely some good code to look at and can base a version of your own. I found AOP perfect for this situation because it does not clutter up the actual function calls. If you have a tiered application, then you can put these performance monitors at each level. (Typically for my webapps, I put it as my data access layer so I can monitor database query performance.)

Answer 4

You could try soap ui if your app is remotely accessible:

http://www.soapui.org/userguide/loadtest/index.html

It'll give you all the sort of stats your after:

http://www.soapui.org/userguide/loadtest/images/loadtest_editor.gif