How does JIT replace optimized machine code during runtime?

Question

I'm browsing through OpenJDK sources and cannot find the place where optimized code is replaced.

I wonder how this can be done in protected mode, isn't it some kind of selfmodifing code which should be prevented by the OS?

Answer 1

The JIT code doesn't replace optimized machine code; it replaces loaded Java bytecode. I don't know how this is implemented in OpenJDK, but typically, the JVM loads the byte code and keeps it in some form of internal structure, usually in a class that has a virtual function or virtual functions for executing the code. When it is just-in-time compiled, the pointer to that internal structure is replaced by a pointer to a class with the same interface, where the underlying representation is native machine code instead of Java byte code, and the virtual methods are implemented such that they invoke the native code rather than interpreting the byte code. There is no modification of code, merely pointing to different places.

Answer 2

The "JITer" allocates space in say the heap or stack and inserts assembly code into it. No, self modifying code is perfectly fine. VirtualProtect (Windows) and mmap (Unix) can map pages as executable. General purpose operating systems by default will mark executable pages as read/execute but not write, you can still typically change this at runtime.

If there was no way to modify code, there would be no way to load a dll unless it's loaded to a fixed Virutal Address and shared into each process's address space; then you'd get address space hell instead of dll hell.

I'm guessing you heard of the NX bit or DEP etc, those just protect you from executing non-executable code, which helps a bit against stack overflows and the likes.