Incremental File Copy Tool and NIH

Question

For years, one of my most important tools has been incremental copy utility that compares the contents of two directories and shows me which files are newer / older / added / deleted. Every day I find myself copying folders of source code between my two desktop machines and the server, and such a utility is critical to avoid overwriting newer files with older ones and also to save time by only copying changed files. In addition, the utility allows me to see new files in the source folder that I don't necessarily want to copy (like temp files) that I instead can delete.

Like anyone who subscribes to the NIH way of thinking, I wrote my own utility to compare the contents of two folders and let me mark files to be copied, deleted, diffed or ignored. I've had many versions of this utility going back to DOS, OS/2 and Win32.

I use this utility on a daily basis, and it leaves me wondering: What do others use? Surely there are similar programs out there to do this... My utility doesn't have a diff screen, and it would be occasionally nice to see what the difference is between two changed files.

What do you use for comparing and incrementally copying between folders?

Answer 1

rsync. All the time.

The biggest benefit to rsync is that it trades increased CPU time for decreased transfer bandwidth, as CPUs are super fast nowadays, and even disk-copy is relatively slow, this is a good thing.

Answer 2

robocopy. It's in Vista, and is also part of the Windows Resource Kit. It has a strange command line interface, but it's very powerful & good for this kind of thing.

Still, I find myself wondering whether source control would be a better choice for you.

Answer 3

SyncToy is also good at this stuff.

Answer 4

I use rsync for some jobs, and unison for others.

For your situation, I would strongly recommend using some version control solution such as Subversion.

As for NIH? While I have written a large number of tools over the years, I always look for an existing tool before writing my own. It saves time, and may have a better solution than I would have used. At the very least, it will give me some "how NOT to do it" examples.

Answer 5

NIH is not necessarily a bad thing. It can be good when the application has some very personal traits and you want it to be as convenient for you as possible, screw the generality.

I've also rolled my own utility (a Perl script) for this purpose a few years ago, and I'm using it both and home and at work for backups. It's simple to run because it does exactly what I need and only that, and simple to tweak because it's written in a flexible scripting language.

Answer 6

SyncBack (free) or SyncBackSE ($$) is another possible solution. SyncBackSE is one of the few programs I've ever paid for. Health warning: Win only.

IMHO, NIH violates Laziness and Impatience, though it strongly supports Hubris.

Answer 7

I have been using rsync (Linux to Linux / WinXP to Linux) a lot and like it. Today I tried to get it up and running under Vista and that was quite a challenge. I got it running but are having some issues with network drives and localized chars (i.e. åäö).

SyncToy seems pretty sweet! I noticed that it puts a data file in the synced folders. Anyone knows if it is possible to use it without the data files / have it save the data files to another folder?

I have to try robocopy as well.

Thanks a lot!

Answer 8

Have you considered using version control tool to accomplish this? It will allow you to keep things in sync while also remembering the history of a project.

Answer 9

I tried Robocopy (included in Vista and available for download for XP) today and it worked fine. To incremental mirror a drive I used: robocopy source destination /MIR

There is also a GUI available: http://technet.microsoft.com/en-us/magazine/cc160891.aspx