STREAM keyword in pig script that runs in Amazon Mapreduce

views:

126

answers:

STREAM keyword in pig script that runs in Amazon Mapreduce

Hi. I have a pig script, that activates another python program. I was able to do so in my own hadoop environment, but I always fail when I run my script in Amazon map reduce WS.

The log say:

org.apache.pig.backend.executionengine.ExecException: ERROR 2090: Received Error while processing the reduce plan: '' failed with exit status: 127 at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:347) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:288) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:260) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:142) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:321) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2216)

Any Idea?

Have you made sure that the script is sent along to the Elastic MapReduce job?

Theo 2010-01-24 14:06:32

No. How can I check it?

sagie 2010-01-24 14:09:06

Not sure, but you could try specifying the path to the script as a full URL including s3:// and all that, like you would with a JAR. See http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/index.html?introduction.html#UsingJarsWithPig

Theo 2010-01-28 08:56:29

Problem solved! All I need is to use the cache('s3://') option when defining the streaming command

sagie 2010-01-24 16:57:21

Would you mind showing your working implementation? I'm in the same boat - I can get a python UDF to run in local mode but it fails when I deploy it to my cluster.

Neil Kodner 2010-07-15 00:35:02

ansaurus

tags:

views:

answers:

STREAM keyword in pig script that runs in Amazon Mapreduce

related questions