Performance creating multiple small files

Question

Hi,

I need a test app that will create a big number of small files on disk as faster as possible.

Will asynch ops help creating files or just writing them? Is there a way to speed up the whole process (writing on a single file is not possible)

Answer 1

Wouldn't physical drive IO be the bottleneck here? You'll probably get different results if you write to a 4200rpm drive versus a 10,000rpm drive versus an ultrafast SSD.

Answer 2

It's hard for me to say without writing a test app myself, but disc access will be synchronized anyway, so it's not like you will have multiple threads writing to disk at the same time. You could speed up your performance by using threads if there was a fair amount of processing done before writing out each file.

Answer 3

If it's possible to test your app using a ramdisk it would probably speed up things considerably.

Answer 4

If possible, don't write them all in the same directory. Many filesystems slow down when dealing with directories containing large numbers of files. (I once brought our fileserver at work, which normally happily serves the whole office, to its knees by writing thousands of files to the same directory).

Instead, make a new directory for each 1000 files or so.