Hadoop 0.2: How to read outputs from TextOutputFormat?

Question

My reducer class produces outputs with TextOutputFormat (the default OutputFormat given by Job). I like to consume this outputs after the MapReduce job complete to aggregate the outputs. In addition to this, I like to write out the aggregated information with TextInputFormat so that the output from this process can be consumed by the next iteration of MapReduce task. Can anyone give me an example on how to write & read with TextFormat? By the way, the reason why I am using TextFormat, rather Sequence, is the interoperability. The outputs should be consumed by any software.

Answer 1

Don't rule out sequence files just yet; they make it fast and easy to chain MapReduce jobs, and you can use "hadoop fs -text filename" to output them in text format if you need them that way for other things.

But, back to your original question: to use TextInputFormat, set it as the input format in the Job, and then use TextInputFormat.setInputPaths to specify what files it should use as input. The key to your mapper should be a LongWritable, and the value a Text.

For using TextOutputFormat as output, set it as the output format in the Job, and then use TextOuputFormat.setOutputPath to specify the output path. Your reducers (or mappers, if it is a map-only job) need to use NullWritable as the type of the output key to get just the text representation of the values one per line, or otherwise each line will be the text representation of the key and the value separated by a tab (by default, you can change this by setting "mapred.textoutputformat.separator" to a different separator).