Benchmarking Performance on Java VM vs .Net CLR

Question

Have you ever had to justify the choice over using .Net instead of Java based on performance?

For a typical high volume transaction processing system that can perform the following operations,

• Concurrent Database transactions
• Mathematical computations
• Interaction with other web services (SOAP/XML, XML-RPC)

My approach would be to code benchmark tests in both Java for the JVM and C# for .Net CLR that benchmark the above operations under various levels of load and compare the results.

Language and platform preferences aside, I am interested in hearing how you would go about doing a conclusive performance comparison between the Java VM and .Net CLR?

Are there any comprehensive and respected benchmarks that exist?

Answer 1

Yeah, there is the benchmark game it is pretty comprehensive and allows you to compare many different things.

http://shootout.alioth.debian.org/

The only thing is it uses mono instead of visual studio but the performance difference between the two is now very small.

in general java is usually slightly faster(depending on what you are doing) but uses a much larger memory footprint and they both are about the same source size.

Answer 2

FYI you are specifically not allowed to use the .net framework code for any form of benchmarking without contacting microsoft first and getting their approval.

If you did want to publish something I thought I'd let you know.

From memory MS did some book store, pet store thing that Java did first to show off how their software could work more effectively at the same task. I'm trying to think of it.

.Net Pet Store

Answer 3

Benchmarks... You can prove almost every thing with test that suites your need. Except Chuck Norris. Chuck Norris is slower and faster than X if he so chooses.

Another point. Let's say you get to conclusion that java is faster. Would that imply that 2 years from now it will still be faster ?

If java is 5% faster and .net is 10% easier to work with what would you choose ? There are many factors and performance is just one of them. And if differences are small (I think they are) it probably is not the most important one.

Unless you are building something that is very performance critical.

Answer 4

I have also wondered which would give better performance. But have no idea about how I might go about performing bench mark test for this.

So good question - hopefully we will all get some guidance here.

I would guess this benchmark testing would need to be a "discount" benchmark testing approach (easy to setup & run by 1 developer)?

I anyone has this kind of info it would be great. I'm often asked to evaluate technologies on my own within short time scales.

Nice one bunn_online!

Answer 5

I don't have exact numbers on the efficiency of the JVM vs the CLR, but the difference, if any, is likely to be small.

However, on the language side, C# does have some more low level constructs than Java, which would allow for more optimization.

Constructs such as:

• User defined value types. Fast to allocate, no memory overhead (which is 12 bytes per reference type in both the CLR and JVM if I remember correctly). Useful for things that let themselves naturally be expressed as values, like vectors and matrices. So mathematical operations. Combine this with ref and out to avoid excessive copying of these large value types.

• Unsafe blocks of code that allow a little more 'close to the metal' optimization. For example, while the CLR and JVM can avoid array bounds checks in some situations, in a lot of cases, they can't and every array access requires a check whether or not the index is still within bounds of the array. Using unsafe code here allows you to access the memory of the array directly with pointers and circumvent any bounds checks. This can mean a significant saving. And on the very low level side, there's also stackalloc which allows you to allocate arrays directly on the stack, whereas a normal array is allocated on the heap, which is slower, but also more convenient. I personally don't know any practical applications of stackalloc.

• True generics, unlike the type erasing generics of Java, avoiding unneeded casting and boxing. But if this is a problem in your Java program, it can easily be solved with some extra work (switching from for example a ArrayList<Integer> to a custom type that internally uses an int[] buffer.)

This all seems biased towards C# and I won't lie: I think it is objectively superior. However, I doubt these differences really matter (and they might not even apply in your case, using pointers gains you nothing if all you do is database access, where Java might be faster) if the choice impedes you in some other way (like going cross platform). Go for correctness, the platform that matches your requirements, rather than minor performance differences.