I've got about 3 million files I need to copy from one folder to another over my company's SAN. What's the best way for me to do this?
+4
A:
Have you considered using rsync? This is a tool that uses an algorithm that involves calculating hashes on chunks of the files to compare two sites and send deltas between the sites.
John Sonmez
2008-12-23 14:21:51
http://samba.anu.edu.au/rsync/
Guillaume
2008-12-23 14:26:35
rsync will only get you a benefit if some of the files already exist at the destination and the destination is a different machine. It would be a win if you had to periodically refresh from one machine to another but won't help for an initial copy.
ConcernedOfTunbridgeWells
2008-12-23 14:38:33
It also won't help you much for copying files from one location on a filesystem on the same machine to another.
ConcernedOfTunbridgeWells
2008-12-23 14:40:03
@NXC: rsync copies between two folders regardless of whether they're on the same machine or not. It doesn't care.
Stewart Johnson
2008-12-23 14:46:09
It doesn't get you any benefit on the same machine (or anywhere) unless you are synchronising two collections. The expensive part of this operation is the disk writes.
ConcernedOfTunbridgeWells
2008-12-23 14:51:51
Yes, but its easy, and the OP didn't say whether it would have to be copied more than once.
John Sonmez
2008-12-23 15:01:55
+1 - the OP says he also has a synchronisation requirement.
ConcernedOfTunbridgeWells
2008-12-23 17:34:58
+6
A:
If a straight copy is too slow (although a SAN with write-back caching would be about as fast as anything for this type of operation) you could tar the files up into one or more archives and then expand the archives out at the destination. This would slightly reduce the disk thrashing.
At a more clever level, you can do a trick with tar or cpio where you archive the files and write them to stdout which you pipe to another tar/cpio process to unravel them at their destination.
A sample command to do this with tar looks like:
tar cf - * | (cd [destination dir] ; tar xf - )
Some SANs will also directly clone a disk volume.
ConcernedOfTunbridgeWells
2008-12-23 14:21:52
A:
If you ask me, its just the best way to copy with neatest system software.
Just something like:
cp -pvr /pathtoolddir /pathtonewdir
on a linux box will do and work great. Any compression in between will just slow down the process.
BeowulfOF
2008-12-23 14:23:53
+3
A:
If you're on windows, use robocopy. It's very robust and build for situations like that. It supports dead link detection and can be told to retry copies if one is interrupted.
Chris
2008-12-23 14:26:00
+2
A:
Microsoft SyncToy is in my experience very good at handling ridiculous numbers of files. And it's very easy to use.
Carl Seleborg
2008-12-23 14:45:45