How to store files on disc to increase performance

Question

From performance point of view: How it's better to store uploaded files on disk?

One folder and huge number of images? or create logically subfolders?

OS: Linux

FS: ext3

Thanks!

Answer 1

From a performance standpoint, using one single folder may or may not be a problem. If you're working with images in the hundreds, a single folder would probably not have a very noticeable difference no matter what. If you have tens or hundreds of thousands of files, it might become an issue. The ext3 filesystem can use indexed directories, meaning it uses an HTree instead of a linked list as the data structure of the filesystem. If it's a linked list, finding a single file or sorting a directory with thousands of files might get slow. If it's an HTree, the performance will probably be similar to a hierarchy of folders. Check and see if your filesystem has the dir_index option (uses HTree for directories) enabled on it: tune2fs -l /dev/hda1 (replace hda1 with your partition). If you want to create a new filesystem with dir_index enabled, pass -O dir_index to the arguments for mke2fs.

Realistically, the file system performance is going to vary a lot depending on your access patterns and how many files there are in the directory. Write a program that you think will have a similar access pattern and try running it on a single directory and then a hierarchy and see what the timing is like.

Answer 2

You probably don't care. You may want to create at least some directories to make the filesystem more managable with commonly-used tools (ls, find etc).

It won't change the performance of opening files very much, but if you really think it matters, try it on production-grade hardware.