How to conduct performance tests on libraries written in different programming languages?

Question

The problem is: Given a number of programming libraries with similar or equal scope (e.g. XML parser, regex, markup, ...); are there tools, with which one can run performance tests on these libraries and compare (and generate reports), even though the libraries may be written in different programming languages (such as java, C#, ruby, python, perl, ...)?

I looked at these opensourcetesting.org/performance.php, but none of them fitted into the (somewhat blurry) requirement above.

Are there toolkits or frameworks out there for cross-language cross-platform performance tests?

Thanks.

Answer 1

I wouldn't try to use a single toolkit for multiple languages. That's unlikely to bring out the best (or even the average) performance for each app.

Instead, I would try to come up with a framework design which defines what it's going to test, and has a common data set. Then each language/library can provide its own framework implementation which tests the operations which are appropriate for that library. That way the operations can be "logically equivalent" even if they don't use the exact same syntax/calls. You end up testing the idiomatic code for that library/language, rather than just a lowest common denominator.

This is the approach I've taken for benchmarking Protocol Buffers. So far my very basic framework has implementations in C# and Java, and I'm now writing a richer framework which allows a whole "benchmark script" to be run. One ideal aim is that different implementations within the same platform (e.g. different .NET implementations of Protocol Buffers) should be able to hook into the same core benchmarking code with very little effort.

The important thing, to my mind, is to have a common set of operations (even if they're not all implemented by all libraries) and a common data set. That's the only way the benchmarks can be meaningful.

Answer 2

If possible, it would be the best to use some form of dependency injection so you can plug out one module and switch it with another. You would need to create unique interface for such libraries before that tho (wrapper that is), which makes a job harder but overall design better. On the positive side, if you want to benchmark earlier interface doesn't have to be complete but just to expose what you think would be most benefitial when perf. is in question.

The abilities very much depend on the language used.

I don't like artificial tests. They are most of the time inacurate IMO, people tend to overlook that one library function does more or less then other library function while they both look like the same thing. I rarely encounter benchmarks that didn't have that kind of property.

On the other sides, there is rarely such a thing as totally better library. Many libraries present good performance in some scenarios where other similar libraries may suck, and vice-versa.

So, if performance is the most important thing for your app, the best thing to do IMO is to create interface for the operations you would like to import, then plugin/out couple of libraries and see the differences in real benchmarks, benchmarks of your app in action, not some artifical math mumbo jumbo...