Would it be possible to implement a custom Thread class in Java (using JNI) in a safe / correct way?
Suppose I write my own NewThread class, with a native start() method, which forks the execution, calls run() in the forked thread and returns...
Is that possible? Would the JVM complain? Is it "legal" according to the specs? Would this break anything, like, in the memory-model? Does it depend on the particular JVM?
It is possible. In the past I was using a C++ lib that read messages for the socket. After the initialization through JNI the library started a couple of pthreads that read data from the socket and did calls in the Java realm through JNI. The said pthreads where venturing pretty deep into the Java code. The only problems we had were memory issue on the JNI seam. After reading the JNI docs and some debugging the issues where solved. So, no problems with memory model.
Don't know if JVM will trigger JIT based on executions incoming from JNI, so could be a performance hit there.
Doable, tricky in places. If you can do with Java Threads, avoid this. I know I would.
Your questions is answered in the Java Native Interface Programmer's Guide and Specification, section 8.1.5.
The important issue is that the VM has to use the same thread model as you are in your native code. Some of the first Java VMs used so called "green threads" on some operating systems (Linux) to emulate thread context switching, since the operating system itself didn't offer native threading support. These "green threads" would not be able to interact with native threads, if you would use one of these old VMs on a newer operating system version with native thread support.
Since Sun's JRE 1.3, I think all "normal" VMs are using native threads directly, which means that you can use native threads yourself in JNI code and expect everything to work as you expect.