How can I benchmark different languages / frameworks?

Question

I'd like to compare the performance of different languages and/or different frameworks within the same language. This is aimed at server-side languages used for web development. I know an apples to apples comparison is not possible, but I'd like it to be as unbiased as possible. Here are some ideas :

• Simple "Hello World" page
• Object initialization
• Function/method calls
  • Method bodies will range from empty to large
• File access (read and write)
• Database access

They can either be measured by Requests per second or I can use a for loop and loop many times. Some of these benchmarks should measure the overhead the language has (ie: empty function call) rather than how fast they perform a certain task. I'll take some precautions:

• They'll run on the same machine, on fresh installations with as few processes on the background as possible.
• I'll try and set up the server as officially recommended; I will not attempt any optimizations.

How can I improve on this?

Answer 1

You will spend a lot of time and come to realization that it was all wasted.
After you complete your tests you will learn that loops of 1000000 empty iterations are far from the real life and come to apache benchmark.
Then you come no know of opcode cachers which will ruin all your previous results.
Then you will learn that single DB query will take 1000 times longer time than API call, so, your comparisons of database access methods are really waste.
Then you will learn of memcache which will allow you just jump over some terrible bottlenecks you've discovered already, etc etc etc

Answer 2

What I have done is to write many unit tests so you can test the layers.

For example, write a SOAP web service in PHP, Python and C#.

Write a REST web service in the same languages (same web services, just two ways to get to them). This one should be able to return JSON and XML as a minimum.

Write unit tests in C# and Python to serve as clients, and test the REST with the various result types (XML/JSON). This is important as later you may need to test to see which is best end-to-end, and JSON may be faster to parse than XML, for you (it should be).

So, the REST/SOAP services should go to the same controller, to simplify your life.

This controller needs tests, as you may need to later remove it's impact on your tests, but, you can also write tests to see how fast it goes to the database.

I would use one database for this, unless you want to evaluate various databases, but for a web test, just do that for phase 2. :)

So, what you end up with is lots of tests, each test needs to be able to determine how long it took for it to actually run.

You then have lots of numbers, and you can start to analyze to see what works best for you.

For example, I had learned (a couple of years ago when I did this) that JSON was faster than XML, REST was faster than SOAP.

You may find that some things are much harder to do in some languages and so drop them from contention as you go through this process.

Writing the tests is the easy part, getting meaningful answers from the numbers will be the harder part, as your biases may color your analysis, so be careful of that.

I would do this with some real application so that the work isn't wasted, just duplicated.

Answer 3

Better take one of the existing benchmarks:

http://www.sellersrank.com/web-frameworks-benchmarking-results/

http://avnetlabs.com/php/php-framework-comparison-benchmarks

http://www.yiiframework.com/performance/

http://www.google.ru/search?q=php+benchmark+frameworks&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:ru:official&client=firefox

But if you really need to find out what framework will be faster for YOUR project - you will need to write a model of your project using that framework and test on it.

Answer 4

There's a lot of good advice (and a huge number of sample benchmarks for different languages) at http://shootout.alioth.debian.org/

C.