Problem with copying local data onto HDFS on a Hadoop cluster using Amazon EC2/ S3.

Question

Hi,

I have setup a Hadoop cluster containing 5 nodes on Amazon EC2. Now, when i login into the Master node and submit the following command

bin/hadoop jar <program>.jar <arg1> <arg2> <path/to/input/file/on/S3>

It throws the following errors (not at the same time.) The first error is thrown when i don't replace the slashes with '%2F' and the second is thrown when i replace them with '%2F':

1) Java.lang.IllegalArgumentException: Invalid hostname in URI S3://<ID>:<SECRETKEY>@<BUCKET>/<path-to-inputfile>
2) org.apache.hadoop.fs.S3.S3Exception: org.jets3t.service.S3ServiceException: S3 PUT failed for '/' XML Error Message: The request signature we calculated does not match the signature you provided. check your key and signing method.

Note:

1)when i submitted jps to see what tasks were running on the Master, it just showed

1116 NameNode
1699 Jps
1180 JobTracker

leaving DataNode and TaskTracker.

2)My Secret key contains two '/' (forward slashes). And i replace them with '%2F' in the S3 URI.

PS: The program runs fine on EC2 when run on a single node. Its only when i launch a cluster, i run into issues related to copying data to/from S3 from/to HDFS. And, what does distcp do? Do i need to distribute the data even after i copy the data from S3 to HDFS?(I thought, HDFS took care of that internally)

IF you could direct me to a link that explains running Map/reduce programs on a hadoop cluster using Amazon EC2/S3. That would be great.

Regards,

Deepak.

Answer 1

You probably want to use s3n:// urls, not s3:// urls. s3n:// means "A regular file, readable from the outside world, at this S3 url". s3:// refers to an HDFS file system mapped into an S3 bucket.

To avoid the URL escaping issue for the access key (and to make life much easier), put them into the /etc/hadoop/conf/core-site.xml file:

<property>
  <name>fs.s3.awsAccessKeyId</name>
  <value>0123458712355</value>
</property>
<property>
  <name>fs.s3.awsSecretAccessKey</name>
  <value>hi/momasgasfglskfghaslkfjg</value>
</property>
<property>
  <name>fs.s3n.awsAccessKeyId</name>
  <value>0123458712355</value>
</property>
<property>
  <name>fs.s3n.awsSecretAccessKey</name>
  <value>hi/momasgasfglskfghaslkfjg</value>
</property>

There was at one point an outstanding issue with secret keys that had a slash -- the URL was decoded in some contexts but not in others. I don't know if it's been fixed, but I do know that with the keys in the .conf this goes away.

Other quickies:

• You can most quickly debug your problem using the hadoop filesystem commands, which work just fine on s3n:// (and s3://) urls. Try hadoop fs -cp s3n://myhappybucket/ or hadoop fs -cp s3n://myhappybucket/happyfile.txt /tmp/dest1 and even hadoop fs -cp /tmp/some_hdfs_file s3n://myhappybucket/will_be_put_into_s3
• The distcp command runs a mapper-only command to copy a tree from there to here. Use it if you want to copy a very large number of files to the HDFS. (For everyday use, hadoop fs -cp src dest works just fine).
• You don't have to move the data to the HDFS if you don't want. You can pull all the source data straight from s3, do all further manipulations targeting either the HDFS or S3 as you see fit.
• Hadoop can become confused if there is a file s3n://myhappybucket/foo/bar and a "directory" (many files with keys s3n://myhappybucket/foo/bar/something). Some old versions of the s3sync command would leave just such 38-byte turds in the S3 tree.
• If you start seeing SocketTimeoutException's, apply the patch for HADOOP-6254. We were, and we did, and they went away.

Answer 2

@mrflip - Thanks for the comments. I tried editing the core-site.xml inside $HADOOP_HOME/conf/ directory. And when i run

hadoop fs -cp s3n://deepak-sample-jar/Twister-Dijstra-0.8.jar source

it throws

cp: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties (respectively).

There is something else i would like to mention. I am using

HADOOP_VERSION = hadoop-0.19.0

in my $HADOOP_HOME/src/contrib/ec2/bin/hadoop-ec2-env.sh (on my workstation). And so, on the cluster, inside /usr/local/hadoop-0.19/conf, there is no core-site.xml. There is hadoop-site.xml instead and i added

fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey

in both the files on all the machines of the cluster (?). But still the problem persists.

I don't know if there is a problem with hadoop-0.19.0. I tried changing HADOOP_VERSION to 0.20.2 but it wouldn't let me login into the cluster. When i run

./hadoop-ec2 launch-cluster <cluster-name> <no.of.nodes>

It throws,

required parameter AMI missing.

Please point me out to a good resource/tutorial on running Map/reduce jobs on a EC2 cluster using Hadoop if you could.

Thanks, Deepak.

Answer 3

Try using Amazon Elastic MapReduce. It removes the need for configuring the hadoop nodes, and you can just access objects in your s3 account in the way you expect.