views:

23

answers:

1

I'm working on a Hadoop streaming workflow for Amazon Elastic Map Reduce and it involves serializing some binary objects and streaming those into Hadoop. Does Hadoop have a maximum line length for streaming input?

I started to just test with larger and larger lines but figured I would ask here first.

A: 

There appears to be no imposed limits on line length. Since asking the question I have been writing code that serializes binary objects, encodes them in base64, then puts them in a stream for processing. As a result, some of the lines are quite long. Hadoop chews right along with no complaints.

JD Long