Hadoop Streaming Multiline Input

Question

I'm using Dumbo for some Hadoop Streaming jobs. I have a bunch of JSON dictionaries each containing an article (multiline text) and some meta data. I know Hadoop performs best when give large files, so I want to concat all the JSON dictionaries into a single file.

The problem is that I don't know how to make Hadoop read each dictionary/article as a separate value instead of splitting on newlines. How can I tell Hadoop to use a custom record separator? Or maybe I can put all of the JSON dictionaries into a list data structure and have Hadoop read that in?

Or maybe encoding the string (base64?) would remove all of the new lines and the normal "reader" would be able to handle it?

Answer 1

You can just replace all newlines with spaecs in each dictionary when concatenating your JSON files. Newline doesn't have any special meaning in JSON besides being a whitespace character.