Retrieving info of a very large directory

Question

Hello,

I hit Linux's 32,000 subdirectory limit. It caused problems with my PHP scripts and I don't want it to happen again.

The simple solution is to have my PHP scripts check the current subdirectory count before trying to create another subdirectory.

All ideas I've seen to perform such a check involve iterating over the entire directory and counting every folder. Considering my concern is with very large directories, is there a better way to retrieve the number of files/folders it contains?

Bonus question: is there also a non-iterative way to find the disk usage of a directory?

Thanks in advance! Brian

Answer 1

A better way is to design your directory layout so there's no way you'll ever have 32000 files in a single directory. In fact, I'd suggest that even 1000 files in a directory is too many.

The approach I usually take to this problem involves extra levels of directory hierarhy. A typical way is to take the file names you're currently storing in a single directory and break it up into pieces that correspond to nested directories. So, if you have a bunch of files like

xyzzy
foo
bar
blah

you might in fact store them as

x/xyzzy
f/foo
b/bar
b/blah

and so on. You can extend this to multiple directory levels, or use more than one character to split the subdirectories, in order to trade off the depth versus breadth of this approach.

You'll probably get suggestions that you use a file system that doesn't have the 32k file limit. Personally, even with such a file system, I would always use a scheme like I'm proposing here. It's almost impossible to work effectively with command-line tools in directories with very very large numbers of files (even ls becomes entirely unwieldy), and this sort of manual exploration is always needed during development, debugging, and often from time to time during normal operation.