Error in Hadoop MapReduce

Question

When I run a mapreduce program using Hadoop, I get the following error.

10/01/18 10:52:48 INFO mapred.JobClient: Task Id : attempt_201001181020_0002_m_000014_0, Status : FAILED java.io.IOException: Task process exit with nonzero status of 1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

10/01/18 10:52:48 WARN mapred.JobClient: Error reading task outputhttp://ubuntu.ubuntu-domain:50060/tasklog?plaintext=true&taskid=attempt_201001181020_0002_m_000014_0&filter=stdout 10/01/18 10:52:48 WARN mapred.JobClient: Error reading task outputhttp://ubuntu.ubuntu-domain:50060/tasklog?plaintext=true&taskid=attempt_201001181020_0002_m_000014_0&filter=stderr

What is this error about?

Answer 1

One reason Hadoop produces this error is when the directory containing the log files becomes too full. This is a limit of the Ext3 filesystem which only allows a maximum of 32000 links per inode.

Check how full your logs directory is in: hadoop/userlogs

A simple test for this problem is to just try and create a directory from the command-line for example: $ mkdir hadoop/userlogs/testdir

If you have too many directories in userlogs the OS should fail to create the directory and report there are too many.