Hadoop ToolRunner fails with NoClassDefFoundError

Question

I am brand new to Linux, Java, and Hadoop. I have a created a simple MapReduce Driver that implements the Tool interface. But when I try to run the job in Eclipse, I get a NoClassDefFoundError before the run() method is invoked.

I am running Hadoop 0.20.2 on Ubuntu 10.04 LTS. The source code and stack trace are provided below. Any assistance will be greatly appreciated.

Thanks, Dave

import org.apache.hadoop.conf.*;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.*;

import org.apache.hadoop.mapreduce.*;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.util.*;

public class MyTestDriver extends Configured implements Tool {

@Override
public int run(String[] args) throws Exception {

    if (args.length != 2) {
        System.err.printf("Usage: %s [generic options] <input> <output>\n",
            getClass().getSimpleName());
        ToolRunner.printGenericCommandUsage(System.err);
        return -1;
    }

    // Code here to submit Hadoop Job ...   
    return 0;
}

/**
 * @param args
 */
public static void main(String[] args) throws Exception {
    int exitCode = ToolRunner.run(new MyTestDriver(), args);
    System.exit(exitCode);
}

}

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at MaxTemperatureDriver.main(MaxTemperatureDriver.java:44) Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.ParseException at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) ... 3 more

Answer 1

Are all of Hadoop's dependencies in your Eclipse build path? Make sure all the jars in the hadoop/lib directory are in your build path.

Answer 2

The error means that there was a particular class that was not found. Classes are stored within .jar files. So, you need to check if all the required jar files are available or not. Jar files are searched based on the CLASSPATH variable. If you are using Eclipse as your development environment, please check if the build dependencies are satisfied (you can do this by - right clicking on your project -> configure build path -> add external jar).

Also, if you are not sure which class is missing, you can check the classname (Hadoop is open source, so you can find the class name) and then search the class name in findjar