Pipeling hadoop map reduce jobs

Question

Hi,

I have five map reduce that I am running each separately. I want to pipeline them all together. So, output of one job goes to next job. Currently, I wrote shell script to execute them all. Is there a way to write this in java? Please provide an example.

Thanks

Answer 1

Hey Algorist,

You may find JobControl to be the simplest method for chaining these jobs together. For more complex workflows, I'd recommend checking out Oozie.

Regards, Jeff

Answer 2

Hi I had similar requirement One way to do this is

after submitting first job execute following

Job job1 = new Job( getConf() );
job.waitForCompletion( true );

and then check for status using

if(job.isSuccessful()){
    //start another job with different Mapper.
    //change config
    Job job2 = new Job( getConf() );
}