Comparing runtimes

Question

I am trying to get some accurate runtime comparisons of PHP vs Python (and potentially any other language that I have to include). Timing within a script is not my problem but timing within a script does not account for everything from the moment the request is made to run the script to output.

1) Is it actually worth taking such things into account? 2) Assuming it is worth taking it into account, how do I do this?

I'm using a Mac so I've got access to Linux commands and I'm not afraid to compile/create a command to help me, I just don't know how to write such a command.

Thanks in advance for any help

Answer 1

If your idea is to compare the languages, I'd say anything outside them is not relevant for comparison purposes.

Nonetheless you can use the time command to measure everything and can compare it with the timing within a script.

Like this:

$ time script.php
HI!

real    0m3.218s
user    0m0.080s
sys     0m0.064s

It will give you clock time, user time (php interpreter) and sys time (OS time)

If you are thinking web, then it gets a lot harder because you would be mixing webserver overhead and that is not always easy to compare if, say, you are using WSGI v/s mod_php. Then you'd have to hook probes into the webserving parts of the chain as well

Answer 2

You can compare performance of different languages using following resource:

Computer Language Benchmarks Game

Answer 3

1. It's worth taking speed into account if you're optimizing code. You should generally know why you're optimizing code (as in: a specific task in your existing codebase is taking too long, not "I heard PHP is slower than Python"). It's not worth taking speed into account if you don't actually plan on switching languages. Just because one tiny module does something slightly faster doesn't mean rewriting your app in another language is a good idea. There are many other factors to choosing a language besides speed.

2. You benchmark, of course. Run the two codebases multiple times and compare the timing. You can use the time command if both scripts are executable from the shell, or use respective benchmarking functionality from each language; the latter case depends heavily on the actual language, naturally.

Answer 4

Well, you can use the "time" command to help:

you@yourmachine:~$ time echo "hello world"
hello world

real    0m0.000s
user    0m0.000s
sys 0m0.000s
you@yourmachine:~$

And this will get around timing outside of the environment.

As for whether you need to actually time that extra work... that entirely depends on what you are doing. I assume this is for some kind of web application of some sort, so it depends on how the framework you use actually works... does it cache some kind of compiled (or parsed) version of the script? If so, then startup time will be totally irrelevant (since the first hit will be the only one that startup time exists in).

Also, make sure to run your tests in a loop so you can discount the first run (and include the cost on the first run in your report if you want). I have done some tests in Java, and the first run is always slowest due to the JIT doing its job (and the same sort of hit may exist in PHP, Python and any other languages you try).