Performance degradation of CreateFile API

Question

Hi, I have an application that is restoring lots of data from a library. We are using Windows server 2003. As the process progresses we are seeing performance degradation. I added some performance counters for 4 of my major tasks, Create file, Read from library, WriteData, Close file. We are restoring millions of files. What I observed is that the performance of createfile API is degrading as the process progresses. The other operations are consistent.

Any ideas what could be causing this?

Thanks in advance for all the answers/suggestions.

Regards, Bhushan

Answer 1

Possibly fighting the system cache? You might look at the Caching Behavior section in the documentation to CreateFile.

Answer 2

Can you provide some information about where the library data is stored, and where the files are being created?

For example, if you are creating all the output files into the same directory you may be finding that CreateFile is slowing down because it needs to check against more and more file names for collisions.

Answer 3

My psychic debugger says you're leaking handles. Are you CloseHandle'ing all of those calls you're CreateFile'ing?

Answer 4

Is the file system NTFS? Are the millions of files going to the same directory, and do they have similar names? From the technet article on NTFS internals:

If you have a large number of files (300,000 or more) in a folder, and the files have long file names with the same initial characters, the time required to create the files increases. The increase occurs because NTFS bases the short file name on the first six characters of the long file name. In folders with more than 300,000 files, the short file names start to conflict after NTFS uses all of the 8.3 names that are similar to the long file names. Repeated conflicts between a generated short file name and existing short file names cause NTFS to regenerate the short file name from 6 to 8 times.

You can query this behavior using the command line utility 'fsutil':

fsutil behavior query disable8dot3