Generating Separate Output files in Hadoop Streaming

Question

Using only a mapper (a Python script) and no reducer, how can I output a separate file with the key as the filename, for each line of output, rather than having long files of output?

Answer 1

Is it possible to replace the outputFormatClass, when using streaming? In a native Java implementation you would extend the MultipleTextOutputFormat class and modify the method that names the output file. Then define your implementation as new outputformat with JobConf's setOutputFormat method

you should verify, if this is possible in streaming too. I donno :-/

Answer 2

You can either write to a text file on the local filesystem using python file functions or if you want to use HDFS use the Thrift API.

Answer 3

The input and outputformat classes can be replaced by use of the -inputformat and -outputformat commandline parameters.

One example of how to do this can be found in the dumbo project, which is a python framework for writing streaming jobs. It has a feature for writing to multiple files, and internally it replaces the output format with a class from its sister project, feathers - fm.last.feathers.output.MultipleTextFiles.

The reducer then needs to emit a tuple as key, with the first component of the tuple being the path to the directory where the files with the key/value pairs should be written. There might still be multiple files, that depends on the number of reducers and the application.

I recommend looking into dumbo, it has many features that makes it easier to write Map/Reduce programs on Hadoop in python.