Get MD5 Checksum for Very Large Files

Question

I've written a script that reads through all files in a directory and returns md5 hash for each file. However, it renders nothing for a rather large file. I assume that the interpreter has some value set for maximum processing time, and since it takes too long to get this value, it just skips along to other files. Is there anyway to get an md5 checksum for large files through PHP? If not, could it be done through a chron job with cpanel? I gave it a shot there but it doesn't seem that my md5sum command has ever been processed: I never get an email with the hash. Here's the PHP I've already written. It's a very simple code and works file for files of a reasonable size:

function md5_dir($dir) {
    if (is_dir($dir)) {
        if ($dh = opendir($dir)) {
            while (($file = readdir($dh)) !== false) {
                echo nl2br($file . "\n" . md5_file($file) . "\n\n");
            }
            closedir($dh);
        }
    }
}

Answer 1

If you're hitting an execution time limit or maximum execution time, PHP should be throwing an error message to that effect. Check your error logs. If you are hitting a limit, you can set the maximum values for PHP memory usage and execution time in your php.ini file:

memory_limit = 16M 

will set max memory usage to 16 megs. For maximum execution time:

max_execution_time = 30

will set maximum execution time to 30 seconds.

Answer 2

While i couldn't reproduce it with PHP 5.2 or 5.3 with a 2GB file the issue seems to come up on 32bit PHP builds.

Even so it's not a really nice solution you could try to let the system to the hasing

echo system("md5sum test.txt");

46d6a7bcbcf7ae0501da341cb3bae27c test.txt

Answer 3

you could achieve it with command line

 shell_exec('md5sum -b '. $fileName);

Answer 4

Make sure to use escapeshellarg ( http://us3.php.net/manual/en/function.escapeshellarg.php ) if you decide to use a shell_exec() or system() call. I.e.,

shell_exec('md5sum -b ' . escapeshellarg($filename));