java.io.IOException: Job failed! when running a sample app on my osx with hadoop-0.19.1

Question

bash-3.2$ echo $JAVA_HOME
/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home
bash-3.2$ bin/hadoop dfs -copyFromLocal conf /user/yokkom/input2
bash-3.2$ bin/hadoop jar hadoop-*-examples.jar grep input2 output 'dfs[a-z.]+'
09/04/17 10:09:32 INFO mapred.FileInputFormat: Total input paths to process : 10
09/04/17 10:09:33 INFO mapred.JobClient: Running job: job_200904171309_0001
java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
        at org.apache.hadoop.examples.Grep.run(Grep.java:69)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.examples.Grep.main(Grep.java:93)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:61)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
        at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)

Do anyone have any idea why this happens? The same job runs perfectly well on linux machines. And, after "Job failed" happens, the whole Hadoop cluster stops responding.

My MacOS version is 10.5.6.

EDIT The same result for hadoop-0.20.1

Answer 1

I googled for it and the same error (at the exact same line) has happened before. Read about it here and here.

The second link suggests the following:

• Delete conf/log4j.properties to see a detailed report
• Try to increase heap memory (using -Xms and -Xmx switches)

Answer 2

Yeah grep can suck up memory, especially on large data sets. How big is your input? BTW, there's also Perm memory.

http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp

There's a few interesting switches to perform custom behaviour on OutOfMemory however I suspect the jobRunner is just catching everything before it gets to the VM. You may need another way to debug it.