How to compare 2 volumes and list modified files?

Question

I have 2 hard-disk volumes(one is a backup image of the other), I want to compare the volumes and list all the modified files, so that the user can select the ones he/she wants to roll-back.

Currently I'm recursing through the new volume and comparing each file's time-stamps to the old volume's files (if they are int the old volume). Obviously this is a blunder approach. It's time consuming and wrong!

Is there an efficient way to do it?

EDIT:
- I'm using FindFirstFile and likes to recurse the volume, and gather info of each file (not very slow, just a few minutes).
- I'm using Volume Shadow Copy to backup.
- The backup-volume is remote so I cannot continuously monitor the actual volume.

Answer 1

Assuming you're not comparing each file on the new volume to every file in the snapshot, that's the only way you can do it. How are you going to find which files aren't modified without looking at all of them?

Answer 2

I am not a Windows programmer. However shouldn't u have stat function to retrieve the modified time of a file. Sort the files based on mod time. The files having mod time greater than your last backup time are the ones of your interest.

For the first time u can iterate over the back up volume to figure out the max mod time and created time from your interested set. I am assuming the directories of interest don't get modified in the backup volume.

Answer 3

Instead of waiting until after changes have happened, and then scanning the whole disk to find the (usually few) files that have changed, I'd set up a program to use ReadDirectoryChangesW to monitor changes as they happen. This will let you build a list of files with a minimum of fuss and bother.

Answer 4

Without knowing more details about what you're trying to do here, it's hard to say. However, some tips about what I think you're trying to achieve:

• If you're only concerned about NTFS volumes, I suggest looking into the USN / change journal API's. They have been around since 2000. This way, after the initial inventory you can only look at changes from that point on. A good starting point for this, though a very old article is here: http://www.microsoft.com/msj/0999/journal/journal.aspx
• Also, utilizing USN API's, you could omit the hash step and just record information from the journal yourself (this will become more clear when/if you look into said APIs)
• The first time through comparing a drive's contents, utilize a hash such as SHA-1 or MD5.
• Store hashes and other such information in a database of some sort. For example, SQLite3. Note that this can take up a huge amount of space itself. A quick look at my audio folder with 40k+ files would result in ~750 megs of MD5 information.

Answer 5

Part of this depends upon how the two volumes are duplicated; if they are 'true' copies from the file system's point of view (e.g. shadow copies or other block-level copies), you can do a few tricky little things with respect to USN, which is the general technology others are suggesting you look into. You might want to look at an API like FSCTL_READ_FILE_USN_DATA, for example. That API will let you compare two different copies of a file (again, assuming they are the same file with the same file reference number from block-level backups). If you wanted to be largely stateless, this and similar APIs would help you a lot here. My algorithm would look something like this:

foreach( file in backup_volume ) {
    file_still_exists = try_open_by_id( modified_volume )
    if (file_still_exists) {
        usn_result = compare_usn_values_of_files( file, file_in_modified_volume )
        if (usn_result == equal_to) {
           // file hasn't changed at all
        } else {
           // file has changed (somehow)
        }
    } else {
        // file was deleted (possibly deleted and recreated)
    }
}
// we still don't know about files new in modified_volume

All of that said, my experience leads me to believe that this will be more complicated than my off-the-cuff explanation hints at. This might be a good starting place, though.

If the volumes are not block-level copies of one another, then it will be very difficult to compare USN numbers and file IDs, if not impossible. Instead, you may very well be going by file name, which will be difficult if not impossible to do without opening every file (times can be modified by apps, sizes and times can be out of date in the findfirst/next queries, and you have to handle deleted-then-recreated cases, rename cases, etc.).

So knowing how much control you have over the environment is pretty important.