Is there a faster alternative to Perl's stat?

Question

I'm traversing an entire partition, stat()'ing each file and then checking the returned values for mtime, size and uid against hashed values. stat() however is far too slow in Perl and I'm wondering if there's any quicker alternatives I may be overlooking.

Answer 1

stat is doing IO on each file which can't be avoided if you're wanting to read those data. So that'll be the limit on speed and can't be worked around any other way that I can think of.

If you're repeatedly stat-ing the same file(s) then consider using Memoize.

use Memoize();

sub fileStat {
  my ($filename) = @_;
  return stat($filename);
}

Memoize::memoize('fileStat');

Answer 2

When you call stat you're querying the filesystem and will be limited by its performance. For large numbers of files this will be slow; it's not really a Perl issue.

Answer 3

Could you post some of your code? I assume that some other optimizations will lie elsewhere. You also might want to try a profiler such as nytprof.

(Hints: map and use an iterative rather than recursive method - this also might be a good use of threads)

Answer 4

Depending on what you're doing with the information, you might be able to generate a find command? However, this still stats every file in the directories to be searched, so the time spent will be similar.

Answer 5

• If you are on *NIX, you can just use ls and parse the output, I should think.
• As Ether mentioned, find is possibly a good alternative if you just want to make decisions on what you stat.
• But size, date, and uid should all be available from ls output.
• While date and size are available from the dir command on a Windows platform.

Answer 6

Before you go off optimizing stat, use Devel::NYTProf to see where the real slow-down is.

Also, investigate the details of how you've mounted the filesystem. Is everything local, or have you mounted something over NFS or something similar? There are many things that can be the problem, as other answers have pointed out. Don't spend too much time focussing on any potential problem until you know it's the problem.

Good luck,

Answer 7

Consider File::Find module.

Answer 8

You've seen that stat is slow enough as it is, so don't call it more than once on the same file.

The perlfunc documentation on -X (the shell-ish file test operators) describes a nice cache for stat:

If any of the file tests (or either the stat or lstat operators) are given the special filehandle consisting of a solitary underline, then the stat structure of the previous file test (or stat operator) is used, saving a system call. (This doesn't work with -t, and you need to remember that lstat and -l will leave values in the stat structure for the symbolic link, not the real file.) (Also, if the stat buffer was filled by an lstat call, -T and -B will reset it with the results of stat _). Example:

print "Can do.\n" if -r $a || -w _ || -x _;
stat($filename);
print "Readable\n" if -r _;
print "Writable\n" if -w _;
print "Executable\n" if -x _;
print "Setuid\n" if -u _;
print "Setgid\n" if -g _;
print "Sticky\n" if -k _;
print "Text\n" if -T _;
print "Binary\n" if -B _;