Hudson fails with error "No space left on device" though there is enough space on the system

Question

Hi, Hudson is repeatedly failing after building few projects with following stack trace with error "No space left on device", though there is enough space on disk. There are no limits of quotas on any folder. Below is output of different system commands.

Here is key system information:

Hudson ver. 1.361
executable-war /opt/hudson/hudson.war
java.runtime.name OpenJDK Runtime Environment
java.runtime.version 1.6.0_18-b18
os.name Linux-Ubuntu 10.04
os.version 2.6.32-19-generic

There is 50% free space according to df

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1            147550696  67382688  72672840  49% /
none                   1535580       304   1535276   1% /dev
none                   1539732       504   1539228   1% /dev/shm
none                   1539732        96   1539636   1% /var/run
none                   1539732         0   1539732   0% /var/lock
none                   1539732         0   1539732   0% /lib/init/rw
none                 147550696  67382688  72672840  49% /var/lib/ureadahead/debugfs

next i checked open file handles by lsof, thats also within limit

lsof | wc -l
694

then i checked file handles by this command

cat /proc/sys/fs/file-nr
3392 0 306935

Detailed Error

[INFO] No tests to run.
[HUDSON] Recording test results
[INFO] [jar:jar {execution: default-jar}]
[INFO] Building jar: /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/target/eif.deployment.CRMServices-1.2-bt.jar
[INFO] [antrun:run {execution: default}]
[INFO] Executing tasks
   [unzip] Expanding: /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/src/it/resources/CRMServices.Driver-soapui-project.zip into /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/src/it/resources
[HUDSON] Archiving /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/pom.xml to /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/modules/rogers.bt.deployment$eif.deployment.CRMServices/builds/2010-07-20_12-13-58/archive/rogers.bt.deployment/eif.deployment.CRMServices/1.2-bt/pom.xml
[HUDSON] Archiving /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/target/eif.deployment.CRMServices-1.2-bt.jar to /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/modules/rogers.bt.deployment$eif.deployment.CRMServices/builds/2010-07-20_12-13-58/archive/rogers.bt.deployment/eif.deployment.CRMServices/1.2-bt/eif.deployment.CRMServices-1.2-bt.jar
[HUDSON] Re-archiving /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/target/eif.deployment.CRMServices-1.2-bt.jar
[INFO] ------------------------------------------------------------------------
[ERROR] FATAL ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to serialize hudson.model.Actionable#actions for class hudson.maven.MavenModuleSetBuild
No space left on device
[INFO] ------------------------------------------------------------------------
[INFO] Trace
java.langchannel stopped
ERROR: Failed to parse POMs
java.io.IOException: Remote call on Channel to Maven [/opt/bea/jdk160_05/bin/java, -cp, /opt/hudson/plugins/maven-plugin/WEB-INF/lib/maven-agent-1.363.jar:/opt/hudson/tools/Maven_2.2.1/boot/classworlds-1.1.jar, hudson.maven.agent.Main, /opt/hudson/tools/Maven_2.2.1, /opt/hudson/war/WEB-INF/lib/remoting-1.363.jar, /opt/hudson/plugins/maven-plugin/WEB-INF/lib/maven-interceptor-1.363.jar, 55951, /opt/hudson/plugins/maven-plugin/WEB-INF/lib/maven2.1-interceptor-1.2.jar] failed
                at hudson.remoting.Channel.call(Channel.java:564)
                at hudson.maven.ProcessCache$MavenProcess.call(ProcessCache.java:156)
                at hudson.maven.MavenModuleSetBuild$RunnerImpl.doRun(MavenModuleSetBuild.java:483)
                at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416)
                at hudson.model.Run.run(Run.java:1253)
                at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:306)
                at hudson.model.ResourceController.execute(ResourceController.java:88)
                at hudson.model.Executor.run(Executor.java:127)
Caused by: java.lang.Error: Unable to load resource hudson/maven/Messages.properties
                at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:198)
                at java.lang.ClassLoader.getResource(ClassLoader.java:977)
                at java.lang.Class.getResource(Class.java:2074)
                at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:83)
                at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:102)
                at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:102)
                at org.jvnet.localizer.ResourceBundleHolder.format(ResourceBundleHolder.java:139)
                at hudson.maven.Messages.MavenBuilder_AsyncFailed(Messages.java:233)
                at hudson.maven.MavenBuilder.call(MavenBuilder.java:184)
                at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:696)
                at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:640)
                at hudson.remoting.UserRequest.perform(UserRequest.java:114)
                at hudson.remoting.UserRequest.perform(UserRequest.java:48)
                at hudson.remoting.Request$2.run(Request.java:270)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
                at java.util.concurrent.FutureTask.run(FutureTask.java:138)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: No space left on device
                at java.io.FileOutputStream.writeBytes(Native Method)
                at java.io.FileOutputStream.write(FileOutputStream.java:247)
                at hudson.remoting.RemoteClassLoader.makeResource(RemoteClassLoader.java:267)
                at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:194)
                ... 19 more
FATAL:  : No space left on device
hudson.util.IOException2:  : No space left on device
                at hudson.XmlFile.write(XmlFile.java:168)
                at hudson.model.Run.save(Run.java:1383)
                at hudson.maven.MavenModuleSetBuild$RunnerImpl.post2(MavenModuleSetBuild.java:595)
                at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528)
                at hudson.model.Run.run(Run.java:1276)
                at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:306)
                at hudson.model.ResourceController.execute(ResourceController.java:88)
                at hudson.model.Executor.run(Executor.java:127)
Caused by: com.thoughtworks.xstream.io.StreamException:  : No space left on device
                at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:73)
                at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.endNode(PrettyPrintWriter.java:288)
                at com.thoughtworks.xstream.io.WriterWrapper.endNode(WriterWrapper.java:37)
                at com.thoughtworks.xstream.io.path.PathTrackingWriter.endNode(PathTrackingWriter.java:48)
                at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:99)
                at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:38)
                at com.thoughtworks.xstream.XStream.marshal(XStream.java:840)
                at com.thoughtworks.xstream.XStream.marshal(XStream.java:829)
                at com.thoughtworks.xstream.XStream.toXML(XStream.java:804)
                at hudson.XmlFile.write(XmlFile.java:165)
                ... 7 more
Caused by: java.io.IOException: No space left on device
                at java.io.FileOutputStream.writeBytes(FileOutputStream.java)
                at java.io.FileOutputStream.write(FileOutputStream.java:260)
                at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
                at java.io.BufferedWriter.flush(BufferedWriter.java:236)
                at hudson.util.AtomicFileWriter.flush(AtomicFileWriter.java:91)
                at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.endNode(PrettyPrintWriter.java:288)
                at com.thoughtworks.xstream.io.path.PathTrackingWriter.endNode(PathTrackingWriter.java:49)
                at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:38)
                at hudson.XmlFile.write(XmlFile.java:165)
                at hudson.model.Run.save(Run.java:1384)
                at hudson.maven.MavenModuleSetBuild$RunnerImpl.post2(MavenModuleSetBuild.java:595)
                at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528) 

Answer 1

The "no space left on device" error isn't necessary caused by running out of storage capacity, as it suggests, it can also be cause by running out of i-nodes on the filesystem. In other words, a given filesystem can only contain so many files. Running df will suggest everything's fine.

See this article for more info.

You either need to delete some files that you don't need any more, or put Hudson on a different filesystem.

It's not uncommon in situations where yo know you'll have a lot of small files to build a filesystem with an explicitly larger inode table.