Here are the arguments that I am using:
-Xmx1024m -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=11999
The -Xmx1024m work fine without passing the -agentlib, and the -agentlib works fine without Xmx. Can you set the max heap size while using the agentlib?
My jar file is getting launched via C code:
execlp(myJavaPath, myJavaPath, myDebugOptions,"-DFBLog4j=true","-jar","myJar.jar", NULL);
Where myDebugOptions contain the char* "-Xmx1024m -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=11999". When I try to run this same command via commandline (in Solaris 10, JVM 1.6.0_17) it works fine.