ansaurus

Question

Hadoop Reduce Error

Answer 1

A:

This link seems to indicate that it could be a network configuration problem.

The reducers are failing to fetch the maps (due to failing too many attempts) which could be due to a networking/configuration problem preventing them being able to query the Jetty server.

Disclaimer: I have not used Hadoop professionally

Syntax 2010-07-13 02:15:15

I already checked all the networking and it doesn't seem to be anything wrong. I also checked the problem when you run out of file descriptors. And I still can't pass this bug. PLEASE HELP! I have to present my thesis in less than three hours.

Marius 2010-07-13 02:27:59

I just discovered that running the wordcount example on the cluster with more than one input file gives the same error. What is the connection?

Marius 2010-07-13 02:32:57

How many nodes in your cluster? Is it possible that somehow you are registering an extra node and when more than one file is submitted the second is given to the additional node? The wordcount example indicates to me that lines are given to Mappers; are there multiple lines in each text file? Meaning that the content of one file would be distributed for processing across multiple hosts?

Syntax 2010-07-13 02:46:58

in the wordcount example I give the sistem some huge texts, with thousands of lines. If the input folder contains just one file everything works out fine, but if a put a copy of the previous text in the same folder and rerun it stops working in the reduce phase with the Exceeded MAX_FAILED_UNIQUE_FETCHES; errorMy sistem has 4 nodes: 1 master and 3 slaves. My project takes advantage of this kind of processing (lines run on different machines)

Marius 2010-07-13 02:52:20

Are you able to test/verify if it works with multiple files with only the master? Then start re-adding the slaves. This may help to identify if/which slave may be causing problems (if it is networking/comms).

Syntax 2010-07-13 02:58:24

I have already tested each node in part at installation. I tried taking out all the nodes and introducing them after but I keep getting some errors about replicas (even thoung in hdfs..xml the number of replicas is 1)

Marius 2010-07-13 03:15:14

ansaurus

tags:

views:

answers:

Hadoop Reduce Error

related questions