IBM Websphere on Windows- OutOfMemoryError: Failed to create a thread

Question

I have a J2EE application running on an IBM Websphere Application Server on a Windows Operating System.

Occasionally I see an OutOfMemoryError Exception with the following information in the javacore file.

1TISIGINFO     Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError":"Failed to create a thread: retVal -1073741830, errno 12" received

Java is run with the following configurations: -Xms512m -Xmx1350m -Xscmx50M

Analyzing the javacore file, the number of threads are just 124. Analyzing the heap dump, the memory occupied by the heap is about 500Mb.

Given the relatively normal number of threads and heap size a lot lower than the maximum, I am trying to figure out why I see this error?

I´m not sure if this helps, but here is the top section of the javacore file...

NULL           ------------------------------------------------------------------------
0SECTION       TITLE subcomponent dump routine
NULL           ===============================
1TISIGINFO     Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError":"Failed to create a thread: retVal -1073741830, errno 12" received 
1TIDATETIME    Date:                 1970/01/01 at 00:00:00
1TIFILENAME    Javacore filename:    d:\WebSphere\AppServer\profiles\AppSrv01\javacore.19700101.000000.652.0003.txt
NULL           ------------------------------------------------------------------------
0SECTION       GPINFO subcomponent dump routine
NULL           ================================
2XHOSLEVEL     OS Level         : Windows Server 2003 5.2 build 3790 Service Pack 2
2XHCPUS        Processors -
3XHCPUARCH       Architecture   : x86
3XHNUMCPUS       How Many       : 2
NULL           
1XHERROR2      Register dump section only produced for SIGSEGV, SIGILL or SIGFPE.
NULL           
NULL           ------------------------------------------------------------------------
0SECTION       ENVINFO subcomponent dump routine
NULL           =================================
1CIJAVAVERSION J2RE 5.0 IBM J9 2.3 Windows Server 2003 x86-32 build j9vmwi3223-20080315
1CIVMVERSION   VM build 20080314_17962_lHdSMr
1CIJITVERSION  JIT enabled - 20080130_0718ifx2_r8
1CIRUNNINGAS   Running as a standalone JVM
1CICMDLINE     d:/WebSphere/AppServer/java/bin/java -Declipse.security -Dwas.status.socket=4434 -Dosgi.install.area=d:/WebSphere/AppServer -Dosgi.configuration.area=d:\WebSphere\AppServer\profiles\AppSrv01/configuration -Dosgi.framework.extensions=com.ibm.cds -Xshareclasses:name=webspherev61,nonFatal -Xscmx50M -Dcom.ibm.nio.DirectByteBuffer.SilentRetry=true -Xbootclasspath/p:d:/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar;d:/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -classpath d:\WebSphere\AppServer\profiles\AppSrv01/properties;d:/WebSphere/AppServer/properties;d:/WebSphere/AppServer/lib/startup.jar;d:/WebSphere/AppServer/lib/bootstrap.jar;d:/WebSphere/AppServer/lib/j2ee.jar;d:/WebSphere/AppServer/lib/lmproxy.jar;d:/WebSphere/AppServer/lib/urlprotocols.jar;d:/WebSphere/AppServer/deploytool/itp/batchboot.jar;d:/WebSphere/AppServer/deploytool/itp/batch2.jar;d:/WebSphere/AppServer/java/lib/tools.jar -Dibm.websphere.internalClassAccessMode=allow -Xms512m -Xmx1350m -Dws.ext.dirs=d:/WebSphere/AppServer/java/lib;d:\WebSphere\AppServer\profiles\AppSrv01/classes;d:/WebSphere/AppServer/classes;d:/WebSphere/AppServer/lib;d:/WebSphere/AppServer/installedChannels;d:/WebSphere/AppServer/lib/ext;d:/WebSphere/AppServer/web/help;d:/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime -Dderby.system.home=d:/WebSphere/AppServer/derby -Dcom.ibm.itp.location=d:/WebSphere/AppServer/bin -Djava.util.logging.configureByServer=true -Duser.install.root=d:\WebSphere\AppServer\profiles\AppSrv01 -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder -Dwas.install.root=d:/WebSphere/AppServer -Dpython.cachedir=d:\WebSphere\AppServer\profiles\AppSrv01/temp/cachedir -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager -Dserver.root=d:\WebSphere\AppServer\profiles\AppSrv01 -Dappserver.platform=was61 -Ddeploymentmgr.rmi.connection=ensi-nd01.sistema-cni.org.br:9809 -Dappserver.rmi.host=ensi-nd01.sistema-cni.org.br -Duser.timezone=GMT-3 -Djava.security.auth.login.config=d:\WebSphere\AppServer\profiles\AppSrv01/properties/wsjaas.conf -Djava.security.policy=d:\WebSphere\AppServer\profiles\AppSrv01/properties/server.policy com.ibm.wsspi.bootstrap.WSPreLauncher -nosplash -application com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer d:\WebSphere\AppServer\profiles\AppSrv01\config ensi-nd01Cell01 ensi-aplic01Node01 lumis4.0.11
1CIJAVAHOMEDIR Java Home Dir:   d:\WebSphere\AppServer\java\jre
1CIJAVADLLDIR  Java DLL Dir:    d:\WebSphere\AppServer\java\jre\bin
1CISYSCP       Sys Classpath:   d:/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar;d:/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar;d:\WebSphere\AppServer\java\jre\lib\vm.jar;d:\WebSphere\AppServer\java\jre\lib\core.jar;d:\WebSphere\AppServer\java\jre\lib\charsets.jar;d:\WebSphere\AppServer\java\jre\lib\graphics.jar;d:\WebSphere\AppServer\java\jre\lib\security.jar;d:\WebSphere\AppServer\java\jre\lib\ibmpkcs.jar;d:\WebSphere\AppServer\java\jre\lib\ibmorb.jar;d:\WebSphere\AppServer\java\jre\lib\ibmcfw.jar;d:\WebSphere\AppServer\java\jre\lib\ibmorbapi.jar;d:\WebSphere\AppServer\java\jre\lib\ibmjcefw.jar;d:\WebSphere\AppServer\java\jre\lib\ibmjgssprovider.jar;d:\WebSphere\AppServer\java\jre\lib\ibmjsseprovider2.jar;d:\WebSphere\AppServer\java\jre\lib\ibmjaaslm.jar;d:\WebSphere\AppServer\java\jre\lib\ibmjaasactivelm.jar;d:\WebSphere\AppServer\java\jre\lib\ibmcertpathprovider.jar;d:\WebSphere\AppServer\java\jre\lib\server.jar;d:\WebSphere\AppServer\java\jre\lib\xml.jar;
1CIUSERARGS    UserArgs:
2CIUSERARG               -Xjcl:jclscar_23
2CIUSERARG               -Dcom.ibm.oti.vm.bootstrap.library.path=d:\WebSphere\AppServer\java\jre\bin
2CIUSERARG               -Dsun.boot.library.path=d:\WebSphere\AppServer\java\jre\bin
2CIUSERARG               -Djava.library.path=d:\WebSphere\AppServer\java\jre\bin;.;D:\WebSphere\AppServer\bin;D:\WebSphere\AppServer\java\bin;D:\WebSphere\AppServer\java\jre\bin;D:\programas\oracle\product\10.2.0\client_1\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;c:\Program Files\Intel\DMIX
2CIUSERARG               -Djava.home=d:\WebSphere\AppServer\java\jre
2CIUSERARG               -Djava.ext.dirs=d:\WebSphere\AppServer\java\jre\lib\ext
2CIUSERARG               -Duser.dir=d:\WebSphere\AppServer\profiles\AppSrv01
2CIUSERARG               _j2se_j9=70912 0x7E7A0BE8
2CIUSERARG               -Dconsole.encoding=Cp850
2CIUSERARG               vfprintf 0x00401145
2CIUSERARG               -Declipse.security
2CIUSERARG               -Dwas.status.socket=4434
2CIUSERARG               -Dosgi.install.area=d:/WebSphere/AppServer
2CIUSERARG               -Dosgi.configuration.area=d:\WebSphere\AppServer\profiles\AppSrv01/configuration
2CIUSERARG               -Dosgi.framework.extensions=com.ibm.cds
2CIUSERARG               -Xshareclasses:name=webspherev61,nonFatal
2CIUSERARG               -Xscmx50M
2CIUSERARG               -Dcom.ibm.nio.DirectByteBuffer.SilentRetry=true
2CIUSERARG               -Xbootclasspath/p:d:/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar;d:/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar
2CIUSERARG               -Dibm.websphere.internalClassAccessMode=allow
2CIUSERARG               -Xms512m
2CIUSERARG               -Xmx1350m
2CIUSERARG               -Dws.ext.dirs=d:/WebSphere/AppServer/java/lib;d:\WebSphere\AppServer\profiles\AppSrv01/classes;d:/WebSphere/AppServer/classes;d:/WebSphere/AppServer/lib;d:/WebSphere/AppServer/installedChannels;d:/WebSphere/AppServer/lib/ext;d:/WebSphere/AppServer/web/help;d:/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
2CIUSERARG               -Dderby.system.home=d:/WebSphere/AppServer/derby
2CIUSERARG               -Dcom.ibm.itp.location=d:/WebSphere/AppServer/bin
2CIUSERARG               -Djava.util.logging.configureByServer=true
2CIUSERARG               -Duser.install.root=d:\WebSphere\AppServer\profiles\AppSrv01
2CIUSERARG               -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder
2CIUSERARG               -Dwas.install.root=d:/WebSphere/AppServer
2CIUSERARG               -Dpython.cachedir=d:\WebSphere\AppServer\profiles\AppSrv01/temp/cachedir
2CIUSERARG               -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager
2CIUSERARG               -Dserver.root=d:\WebSphere\AppServer\profiles\AppSrv01
2CIUSERARG               -Dappserver.platform=was61
2CIUSERARG               -Ddeploymentmgr.rmi.connection=ensi-nd01.sistema-cni.org.br:9809
2CIUSERARG               -Dappserver.rmi.host=ensi-nd01.sistema-cni.org.br
2CIUSERARG               -Duser.timezone=GMT-3
2CIUSERARG               -Djava.security.auth.login.config=d:\WebSphere\AppServer\profiles\AppSrv01/properties/wsjaas.conf
2CIUSERARG               -Djava.security.policy=d:\WebSphere\AppServer\profiles\AppSrv01/properties/server.policy
2CIUSERARG               -Dinvokedviajava
2CIUSERARG               -Djava.class.path=d:\WebSphere\AppServer\profiles\AppSrv01/properties;d:/WebSphere/AppServer/properties;d:/WebSphere/AppServer/lib/startup.jar;d:/WebSphere/AppServer/lib/bootstrap.jar;d:/WebSphere/AppServer/lib/j2ee.jar;d:/WebSphere/AppServer/lib/lmproxy.jar;d:/WebSphere/AppServer/lib/urlprotocols.jar;d:/WebSphere/AppServer/deploytool/itp/batchboot.jar;d:/WebSphere/AppServer/deploytool/itp/batch2.jar;d:/WebSphere/AppServer/java/lib/tools.jar
2CIUSERARG               vfprintf
2CIUSERARG               _port_library 0x7E7A04F8
2CIUSERARG               -Xdump
NULL

Answer 1

It seems like you run out of native memory. See "Thanks for the memory".

Reduce the heap space to 1024m:

2CIUSERARG               -Xms512m
2CIUSERARG               -Xmx1350m

Do you still have this OutOfMemoryExceptions?

Answer 2

is this changes solving this issue ? i've same issue with this, changing from 1500m -> 1200m then 900m , doesnt solved issue ? running thread only 150-250. Do you have any idea ?