Can I get the MD5sum of a directory with Perl?

Question

I am writing a Perl script (in Windows) that is using File::Find to index a network file system. It works great, but it takes a very long time to crawl the file system. I was thinking it would be nice to somehow get a checksum of a directory before traversing it, and it the checksum matches the checksum that was taken on a previous run, do not traverse the directory. This would eliminate a lot of processing, since the files on this file system do not change often.

On my AIX box, I use this command:

csum -h MD5 /directory

which returns something like this:

5cfe4faf4ad739219b6140054005d506  /directory

The command takes very little time:

time csum -h MD5 /directory
5cfe4faf4ad739219b6140054005d506  /directory

real    0m0.00s
user    0m0.00s
sys     0m0.00s

I have searched CPAN for a module that will do this, but it looks like all the modules will give me the MD5sum for every file in a directory, not for the directory itself.

Is there a way to get the MD5sum for a directory in Perl, or even in Windows for that matter as I could call a Win32 command from Perl?

Thanks in advance!

Answer 1

In order to get a checksum you must read the files, this means you will need to walk the filesystem, which puts you back in the same boat you are trying to get out of.

Answer 2

Can you just read the last modified dates of the files and folders? Surely that's going to be faster than building MD5's?

Answer 3

From what I know you cannot get an md5 of a directory. md5sum on other systems complains when you provide a directory. csum is most likely giving you a hash of the directory file contents of the top level directory, not traversing the tree.

You can grab the modified times for the files and hash them how you like by doing something like this:

sub dirModified($){
    my $dir = @_[0];
    opendir(DIR, "$dir");
    my @dircontents = readdir(DIR);
    closedir(DIR);

    foreach my $item (@dircontents){
        if( -f $item ){
            print -M $item . " : $item - do stuff here\n";
        } elsif( -d $item && $item !~ /^\.+$/ ){
            dirModified("$dir/$item");
        }
    }
}

Yes it will take some time to run.

Answer 4

In addition to the other good answers, let me add this: if you want a checksum, then please use a checksum algorithm instead of a (broken!) hash function.

I don't think you don't need a cryptographically secure hash function in your file indexer -- instead you need a way to see if there are changes in the directory listings without storing the entire listing. Checksum algorithms do that: they return a different output when the input is changed. They might do it faster since they are simpler than hash functions.

It is true that a user could change a directory in a way that wouldn't be discovered by the checksum. However, a user would have to change the file names like this on purpose since normal changes in file names will (with high probability) give different checksums. Is it then necessary to guard against this "attack"?

One should always consider the consequences of each attack and choose the appropriate tools.

Answer 5

I did one of these in python if your interested:

http://akiscode.com/articles/sha-1directoryhash.shtml