A way to make md5_file() faster?

Question

I currently use md5_file(); to run through about 15 URLs and verify their MD5. Is there a way I can make this faster? It takes far too long to run through all of them. Sorry if this isn't a good question, I just realized its only three (four if you count this one) sentences long.

Answer 1

The md5 algorithm is pretty much as fast as it can get, and fetching urls is pretty much as fast as it can get (slow if the files are huge or you have a slow connection). So no. You can't make it faster.

Answer 2

Well obviously you can not do anything with md5_file() to make faster, however, you can use some micro-optimizations or code re-factoring to get some speed gain but again you can not speed up the built-in function md5_file().

Answer 3

No. Since this is a built in function there's no way to make it faster.

But if your code is downloading files before MD5ing them, it may be possible to optimize your downloads to be faster. You may also see a small speed increase by setting the size of the file (using ftruncate) before writing it if you know the size ahead of time.

Also, if the files are small enough to hold in memory and you already have them in memory (because they have been downloaded, or are being read for some other purpose) then you could use md5 to operate on it in memory rather than md5_file which requires it be read again from the disk.

Answer 4

Probably you're doing it sequentially right now. I.e. fetch data 1, process data1, fetch data 2, process data 2, ... and the bottleneck might be the data transfer.
You could use curl_multi_exec() to parallelize that a bit. Either register a CURLOPT_WRITEFUNCTION and process each chunk of data (tricky since md5() works on exactly one chunk of data).
Or check for curl handles that are already finished and then process the data of that handle.

edit: quick&dirty example using the hash extension (which provides functions for incremental hashes) and a php5.3+ closure:

$urls = array(
  'http://stackoverflow.com/',
  'http://sstatic.net/so/img/logo.png',
  'http://www.gravatar.com/avatar/212151980ba7123c314251b185608b1d?s=128&d=identicon&r=PG',
  'http://de.php.net/images/php.gif'
);

$data = array();
$fnWrite = function($ch, $chunk) use(&$data) {
  foreach( $data as $d ) {
    if ( $ch===$d['curlrc'] ) {
      hash_update($d['hashrc'], $chunk);
    }
  }
};

$mh = curl_multi_init();
foreach($urls as $u) {
  $current = curl_init();
  curl_setopt($current, CURLOPT_URL, $u);
  curl_setopt($current, CURLOPT_RETURNTRANSFER, 0);
  curl_setopt($current, CURLOPT_HEADER, 0);
  curl_setopt($current, CURLOPT_WRITEFUNCTION, $fnWrite);
  curl_multi_add_handle($mh, $current);
  $hash = hash_init('md5');
  $data[] = array('url'=>$u, 'curlrc'=>$current, 'hashrc'=>$hash); 
}

$active = null;
//execute the handles
do {
  $mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);

while ($active && $mrc == CURLM_OK) {
  if (curl_multi_select($mh) != -1) {
    do {
      $mrc = curl_multi_exec($mh, $active);
    } while ($mrc == CURLM_CALL_MULTI_PERFORM);
  }
}

foreach($data as $d) {
  curl_multi_remove_handle($mh, $d['curlrc']);
  echo $d['url'], ': ', hash_final($d['hashrc'], false), "\n";
}
curl_multi_close($mh);

(haven't checked the results though ...it's only a starting point)

Answer 5

Presumably you are checking the same URLs over a period of time? Could you check the last modified headers for the URL? If the page being checked has not changed then there would be no need to re-compute the MD5.

You could also request the pages asynchronously so they could be processed in parallel, rather than in serial, which should speed it up.

Answer 6

The speed of the MD5 algorithm is linear. The bigger the input, the more time it will take, so if the file is big, there's not much you can do, really.

Now, as VolkerK already suggested, the problem is most likely not the md5 hashing but retrieving and reading the file over the net.

Answer 7

I see a very good suggestion of optimizing here. This will work well especially for big files, where md5_file is reading the file and this function is just comparing the second byte of each file.