What are the downsides of JIT compilation?

Question

Just In Time Compilation has a number of theoretical advantages: it has more information about the machine, the state of flags, and how the code is used. It allows you to avoid time-consuming compilation before running, speeding up the development cycle.

Simplistically, JIT compilation seems superior as an approach; there is some overhead, but if Sufficiently Smart it can speed up your code enough to offset that.

However, I don't think that's the whole story. What are the theoretical and practical downsides? Yes, "slower startup time" is often mentioned, as is "increased memory consumption", but I wonder if there is more than that.

For example, does tracing and JIT compilation trash the cpu cache? If your program is large and doesn't really have many particularly hot paths, is there a risk of spending more time tracing and JIT-ting than would be worthwhile?

It seems likely that someone has written a paper on this or on solving problems inherent in JIT. If someone has measurements, all the better.

Edit: I'm talking about Just In Time compilation in comparison to ahead of time compilation (potentially with feedback directed optimization), not in comparison to interpretation.

Answer 1

If the JIT is working properly, there is no real disadvantage. Having said that, it's taken Sun a long time to stabilize Hotspot because it's extremely complex. With regards to benchmark data, you can always run the following experiment:

Run SPECjbb, SPECjvm, or your own benchmark and modify the command line that executes java to include:

-Xint

This will exclude any runtime compilation from occuring.

Answer 2

The main problem is that currently the VM must load--this is loading an entire compiler and runtime into memory so it's not going to be quick.

Once it's loaded, compiling and profiling can be done in the background over a long term and can theoretically be significantly faster than any statically compiled language by hand-optimizing to fit the runtime situation. (Think of the case where a function is determined to have no side-effects and not rely on outside data that is called 50 times a second with the same parameters--a dynamically compiled language could simply learn to return a constant. This is something a statically compiled language simply cannot do generically.)

Note that this isn't necessarily always going to be a problem with VMs, if they start building the VMs into the OS and reusing VMs across multiple apps, this problem would go away.

Also vm bytecode tends to do much more with much less bytes. This is why Microsoft initially used VMs in Excel/Word (pre-java), it was simply to reduce code size.

This part is just conjecture, but I'd think this would mean that a custom CPU could be optimized to operate more quickly because reading/writing would be less of a bottleneck. RISC systems tend to assume that the CPU operations and not the I/O are the bottleneck, I don't believe this is always true--doing more work in a single "opcode" should mean more opportunities for the CPU manufactures to optimize their hardware.

My point? There are no innate drawbacks, and the main real-world drawback is load-time.

Oh, another real-world drawback, Java and C# tend to allocate all objects on the heap. Heap allocations are much quicker in C#/Java than with C++, but still not up to stack allocation speeds.

Answer 3

For example, does tracing and JIT compilation trash the cpu cache? If your program is large and doesn't really have many particularly hot paths, is there a risk of spending more time tracing and JIT-ting than would be worthwhile?

That's plausible. But the whole optimization game is about trading off various factors to achieve the best result in the average case. Most applications do have relatively hot and cold paths, even if the hot paths are all in the standard class libraries.

Besides what you are effectively saying is that this hypothetical application is not worth JIT compiling at all. In that case, the "fix" would be to run it with JIT compilation turned off.

It seems likely that someone has written a paper on this or on solving problems inherent in JIT.

You'd have thought so.

But the flip-side is that the folks who build and maintain the JIT compilers in the Oracle, IBM, etc JVMs may be restricted from telling the world about their ideas and results ... for commercial reasons. (Publishing the source code is one thing, but explaining why they chose a particular strategy is something else.) There's also the issue of motivation.