web programming language benchmarks?

Question

I'm thinking that as more and more programming moves to the web with similar and a variety of tools and techniques, has anyone come up with a good set of benchmarks for a programming language?

I'm thinking server-side languages, like Java, PHP, ASP.net, and C# are the big ones. Though it could be anything like C or perl, I don't want to go there because it's not mainstream or even fast.

When I buy a new CPU, I can benchmark the floating point, integer, and multi-task numbers. I can bench things like rendering times.

I would think someone could develop a standard bench for simple programming tasks, and practical web serving scenarios. Preferably an independent organization. Of course you can always tweak code for performance, but I am talking simple things that can be done identically in every language. I bet it could be robust as well.

At some point for scalability, performance will be an issue for web languages. I would like to have a way to tell if a language, or better yet, the latest release of it, stacks up against the competition.

Anybody doing this yet, or should I come up with my own business?

EDIT: yes I know benchmarks are subjective, even more-so for web pages. I didn't say it would be easy, or even necessary now.. Clearly benchmarking my video card on Crysis versus Far Cry is subjective, but it is a useful real world test. That's what I'm aiming for, and for the mainstream languages. Could we see a resurgence in C if performance becomes a focus point down the road?

Answer 1

Are you referring to computational speed or programming productivity? Or perhaps something else?

Aside from actual number crunching speed, language quality is highly subjective, and there are those who would argue that programming productivity is more important than computing speed.

Furthermore, while many languages have common characteristics, each language has specific goals. It would be like comparing apples to oranges.

Answer 2

There are some benchmarks out there comparing a lot of server side languages.

I've always enjoyed fractals, and was curious if scripting languages were up to the task. I wrote a very simple Mandelbrot set generator for my test. Rather than optimizing for each language, I tried to write each program in approximately the same way in each language to make a reasonable performance comparison.

Here are the results from running on my 867 mhz Powerbook G4. Shorter is better. Please note, the following benchmarks are not scientific, and were simply done to satisfy my curiosity. Your mileage may vary.

Feel free to send me ports to any other languages. The program should print the time in seconds that elapsed at the bottom in the form of 'Elapsed %0.2f'. If you can, include instructions for building on MacOS X.

Language                Time    Relative Speed
C gcc-4.0.1             0.05 seconds     1.00 x
ocaml compiled 3.09.2   0.05 seconds     1.00 x
SBCL 1.0.2              0.13 seconds     2.55 x
Java 1.4.2              0.40 seconds     8.00 x
Io 20070410 Vector      1.40 seconds     28.09 x
Lua 5.1                 1.50 seconds     30.00 x
ocaml bytecode 3.09.2   3.76 seconds     75.15 x
Python 2.5.1            9.99 seconds     199.80 x
Ghostscript 8.51        11.66 seconds     233.12 x
Perl 5.8.6 Optimized    12.37 seconds     247.34 x
TCL 8.4 Optimized       16.00 seconds     320.00 x
Perl 5.8.6              21.75 seconds     435.00 x
PHP 5.1.4               23.12 seconds     462.40 x
Javascript SpiderMonkey v1.6 31.06 seconds     621.27 x
Ruby 1.8.4              34.31 seconds     686.18 x
Emacs Lisp              47.25 seconds     945.00 x
Applescript             71.75 seconds     1435.00 x
Io 20070410             85.26 seconds     1705.13 x

Answer 3

The floating point etc figures from a CPU are less subjective than a web page / webservice etc. to make a fair comparison between languages will be difficult and often the problem is how it is written, not the language itself. It would be difficult to accurately compare languages, although not impossible.

The easiest analogy for me is the database TPC scoring mechanisms are well known and do provide performance figures on the database side but when does the average developer even go look at them? Unless you are not working at an extreme level they are not your guiding factor (and even then, the DB's include optimiser shortcuts to make the TPC scores better.)

TPC made a non-profit business out of it, but it takes significant buy in from the competing products - if MS / Oracle / IBM along with the hardware vendors did not buy into it, it would not of got very far.

Could it be done? possibly, never say never, but it wouldn't be trivial at all and requires buy-in from multiple language vendors, h/w vendors etc. It's more about sales and marketting than technology at that point.

Answer 4

has anyone come up with a good set of benchmarks for a programming language?

Probably not and this seems increasingly less likely to happen - see "Flawed Benchmarks" and particularly "Benchmarks are a crock".

I am talking simple things that can be done identically in every language

What makes you think you can tell that they are "done identically in every language"? The program source code might look the same but that doesn't mean the same thing is done by the language interpreters or the same processor instructions are generated by the compilers.