Hello,
I get multiple small files into my input directory which I want to merge into a single file without using the local file system or writing mapreds. Is there a way I could do it using hadoof fs commands or Pig?
Thanks!
A:
Thanks Harsha, But can the destination file for getmerge be within the DFS? From what I understand, the destination has to be the local file system
uHadoop
2010-08-25 04:44:59
A:
okay...I figured out a way using hadoop fs commands -
hadoop fs -cat [dir]/* | hadoop fs -put - [destination file]
It worked when I tested it...any pitfalls one can think of?
Thanks!
uHadoop
2010-08-25 08:49:08
A:
You can use the tool HDFSConcat, new in HDFS 0.21, to perform this operation without incurring the cost of a copy.
Jeff Hammerbacher
2010-10-04 11:46:06
Thanks Jeff, will look into HDFSConcat. Currently we are on 0.20.2 so I am now creating a Har of all the files and then reading from pig. This way data stays in HDFS.
uHadoop
2010-10-04 11:52:36
I should note that this tool has limitations highlighted at https://issues.apache.org/jira/browse/HDFS-950. Files must have the same block size and be owned by the same user.
Jeff Hammerbacher
2010-10-05 09:23:24