What programming language implementation benchmarks are there?

Question

I remember from my old BASIC programming days that for extra performance, you could use a BASIC compiler (instead of the runtime interpreter), but for ultimate performance, the answer was assembly code.

Today, when performance is crucial, I guess assembly language is still the ultimate optimisation answer. Going for a less extreme solution, many seem to write parts of their application in C/C++ to optimise bottlenecks.

This leads me to wonder what "benchmark tests" are published on the performance of various programming language implementations. I guess such tests will be most "truthful" when comparing implementations of the same programming language, but they could also provide some insight on how much is to be gained in speed by writing parts of an application in a different programming language.

Some nice references have been given here, but they are not very recent. I would be interested to see more comparisons of .NET and Java implementations.

Answer 1

Yes, there are lots and lots. It really depends what kind of area you're talking about. You can start at http://dan.corlan.net/bench.html or http://shootout.alioth.debian.org/ . Also keep in mind that writing in assembly only helps if you're better at assembly than the compiler(-writers); this is definitely not always true. This also holds for other domains by analogy.

Answer 2

There're lies, damned lies and benchmarks.

Answer 3

I dont have the references at hand, but I remember reading an article that benchmarked hand-written assembly, C, C++ and Java. For most cases (except the simplest ones) the C compiler was better than hand-written assembly. For quite a few cases Java also had better performances than C.

Modern CPUs are complex enough that it is pretty difficult to really optimize your code by hand. Hand running your code on a VM opens a whole class of optimization that's not available to a compiler. Like choosing specialized instructions that are available only to a specific architecture ...

As always, the only way to be sure is to benchmark, find bottlenecks, optimize them and benchmark again. Most of the time performance issues dont come from where we expect them. And most of the time, improvements come more from better algorithm than from better implementations ...

Answer 4

Matthew Flaschen > You can start at http://dan.corlan.net/bench.html

Copyright (c) 2003