In terminal, merging multiple folders into one.

Question

I have a backup directory created by WDBackup (western digital external HD backup util) that contains a directory for each day that it backed up and the incremental contents of just what was backed up.

So the hierarchy looks like this:

20100101
  My Documents
    Letter1.doc
  My Music
    Best Songs Every
      First Songs.mp3
    My song.mp3 # modified 20100101
20100102
  My Documents
    Important Docs
      Taxes.doc
  My Music
    My Song.mp3 # modified 20100102

...etc...

Only what has changed is backed up and the first backup that was ever made contains all the files selected for backup.

What I'm trying to do now is incrementally copy, while keeping the folder structure, from oldest to newest, each of these dated folders into a 'merged' folder so that it overrides the older content and keeps the new stuff.

As an example, if just using these two example folders, the final merged folder would look like this:

Merged
  My Documents
    Important Docs
      Taxes.doc
    Letter1.doc
  My Music
    Best Songs Every
      First Songs.mp3
    My Song.mp3 # modified 20100102

Hope that makes sense.

Thanks,

Josh

Answer 1

You can use rsync in a bash for loop, e.g.:

$ for d in 2010* ; do rsync -av ./$d/ ./Merged/ ; done

Note that before running this for real you might just want to be cautious and test that it's actually going to do what you want it to - for this you can use rsync's -n flag to do a "dry run":

$ for d in 2010* ; do rsync -avn ./$d/ ./Merged/ ; done

Answer 2

If directory names are actually in the YYYYMMDD form you can just sort 'em. So it's just a matter of traversing them one at a time starting from the older and copying everything to the destination, overwriting previous stuff. Pretty inefficient, but works.

cd $my_disk_full_of_backups
for x in (ls | sort); do cp -Rf $x/* /my_destination/; done

Ugly as it is, should also be fairly portable on nearly every unix system that has bash.

See bash(1), ls(1) and sort(1) for details.