JNI_ENOMEM from JNI_CreateJavaVM when calling dll that uses JNI from VB6

Question

I work on a legacy system that has a VB6 app that needs to call Java code. The solution we use is to have the VB app call a C++ dll that uses JNI to call the Java code. A bit funky, but it's actually worked pretty well. However, I'm moving to a new dev box, and I've just run into a serious problem with this. The built VB app works fine on the new box, but when I try to run it from VB, the dll fails to load the VM, getting a return code of -4 (JNI_ENOMEM) from JNI_CreateJavaVM.

Both the built app and VB are calling the exact same dll, and I've tried it with both Java 1.5 and 1.6. I've tried the suggestions here (redirecting stdout and stderr to files, adding a vfprint option, adding an -Xcheck:jni option), but to no avail. I can't seem to get any additional information out of the jvm. As far as I can tell, the new box is configured pretty much the same as the old one (installed software, Path, Classpath, etc.), and both are running the same release of Windows Server 2003. The new machine is an x64 box with more memory (4GB rather than 2GB), but it's running 32-bit Windows.

Any suggestions or ideas about what else to look into? Rewriting the whole thing in a more sane way is not an option -- I need to find a way to have the dll get the jvm to load without thinking that it's out of memory. Any help would be much appreciated.

Answer 1

OK, I've figured it out. As kschneid points out, the JVM needs a pretty large contiguous chunk of memory inside the application's memory space. So I used the sysinternals VMMap utility to see what VB's memory looked like. There was, in fact, no large chunk of memory available, and there were some libraries belonging to Visio that were loadeed in locations that seemed to be designed to fragment memory. It turns out that when I installed Visio on the new machine, it automatically installed the Visio UML add-in into VB. Since I don't use this add-in, I disabled it. With the add-in disabled, there was a large contiguous chunk of free memory available, and now the JVM loads just fine.

Answer 2

I had the same problem described by "the klaus" and read "http://support.microsoft.com/kb/126962". Changed the registry as described in the mentioned article. I exagerated my change to : "%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=3072,3072,3072 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off MaxRequestThreads=16"

The field to look at is "SharedSection=3072,3072,3072". It solved my problem, but I may have side-effects because of this change.