Why am I getting this java.io.exception "Can't allocate memory" from Hudson?

Question

Here's what I believe to be the relevant error message:

Caused by: java.io.IOException: Cannot run program "/usr/bin/git" (in directory "/var/lib/hudson/jobs/Goals/workspace"): java.io.IOException: error=12, Cannot allocate memory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:474)
    at hudson.Proc$LocalProc.<init>(Proc.java:192)
    at hudson.Proc$LocalProc.<init>(Proc.java:164)
    at hudson.Launcher$LocalLauncher.launch(Launcher.java:638)
    at hudson.Launcher$ProcStarter.start(Launcher.java:273)
    at hudson.Launcher$ProcStarter.join(Launcher.java:280)
    at hudson.plugins.git.GitAPI.launchCommandIn(GitAPI.java:319)
    ... 15 more
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
    at java.lang.ProcessImpl.start(ProcessImpl.java:81)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:467)
    ... 21 more

You can also see the full output from Hudson here: http://hudson.pastebin.com/KLSNrY1D

Any ideas? How can I stop this from happening? I'm not a Java dev, so I don't know much about what's going on here. I have to restart Hudson entirely to fix the issue, but clearly this is not the best answer.

Answer 1

How many external programs do you run at a time? How many Hudson builders, etc? What does "free" say when this happens?

You might simply have run into that the system has run out of memory available to the hudson process with children.

Answer 2

Sounds like you've exhausted swap space on your box.

The java.lang.ProcessBuilder.start() ultimately must boil down to a fork or clone system call on a Unix-like OS to create a new process. That takes swap space. And you seem to not have enough. This is more in the Unix system admin realm, not Java.