Win32 IO Performance Problem

Question

Recently I ran into a "fun" problem with the Microsoft implementation of the CRTL. tmpfile places temp files in the root directory and completely ignores the temp file directory. This has issues with users who do not have privileges to the root directory (say, on our cluster). Moreover, using _tempnam would require the application to remember to delete the temporary files, which it is unable to do without a considerable amount of rework.

Therefore I bit the bullet and wrote Win32 versions of all of the IO routines (create_temp, read, write, seek, flush) which call the appropriate method. One thing I've noticed is the now abysmal performance of the library.

Results from the test suite:

CRTL:    4:30.05 elapsed
Win32:  11:18.06 elapsed

Stats measured in my routines:
Writes:  3129934 (   44,642,745,008 bytes)
Reads:    935903 (    8,183,423,744 bytes)
Seeks:   2205757 (2,043,782,657,968 bytes traveled)
Flushes:   92442

Example of a CRTL v. Win32 method:

int io_write(FILE_POINTER fp, size_t words, const void *buffer)
{
#if !defined(USE_WIN32_IO)
    {
        size_t words_written = 0;

        /* read the data */
        words_written = fwrite(buffer, sizeof(uint32_t), words, fp);
        if (words_written != words)
        {
            return errno;
        }
    }
#else /* !defined(USE_WIN32_IO) */
    {
        DWORD bytesWritten;

        if (!WriteFile(fp, buffer, words * sizeof(uint32_t), &bytesWritten, NULL)
            || (bytesWritten != words * sizeof(uint32_t)))
        {
            return GetLastError();
        }
    }
#endif /* USE_WIN32_IO */

    return E_SUCCESS;
}

As you can see, they are effectively identical, yet the performance (in release mode) is wildly divergent. Time spent in WriteFile and SetFilePointer dwarf the time spent in fwrite and fseeko, which seems counterintuitive.

Ideas?

UPDATE: perfmon notes that fflush is about 10x cheaper than FlushFileBuffers and fwrite is ~1.1x slower than WriteFile. The net result is a huge performance loss with FlushFileBuffers used in the same manner as fflush. There is no change from FILE_ATTRIBUTE_NORMAL to FILE_FLAG_RANDOM_ACCESS either.

Answer 1

Traditionally, the C runtime library functions buffer the data and only trigger the write operation (hence the need for functions like fflush). I don't think that WriteFile buffers the write operation so every time you call WriteFile, an I/O operation gets triggered whereas with fwrite, the I/O gets triggered when the buffer has reached a certain size.

As you can see from your measurements, the buffered I/O tends to be more efficient...

Answer 2

I think it's probably due to this issue, described on MSDN's page for FlushFileBuffers:

Due to disk caching interactions within the system, the FlushFileBuffers function can be inefficient when used after every write to a disk drive device when many writes are being performed separately. If an application is performing multiple writes to disk and also needs to ensure critical data is written to persistent media, the application should use unbuffered I/O instead of frequently calling FlushFileBuffers. To open a file for unbuffered I/O, call the CreateFile function with the FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH flags. This prevents the file contents from being cached and flushes the metadata to disk with each write. For more information, see CreateFile.

In general, FlushFileBuffers is an "expensive" operation, since it flushes everything in the write-back cache:

FlushFileBuffers(): This function will flush everything in the write-back cache, as it does not know what part of the cache belongs to your file. This can take a lot of time, depending on the cache size and the speed of the media. How necessary is it? There is a thread which goes through and writes out dirty pages, so it is likely not very necessary.

I presume that fflush does not flush the entire write-back cache. In that case, it's much more efficient, but that efficiency comes at the risk of potential data loss. The CRT's source code for fflush confirms this, since _commit calls FlushFileBuffers:

/* lowio commit to ensure data is written to disk */
if (str->_flag & _IOCOMMIT) {
        return (_commit(_fileno(str)) ? EOF : 0);
}

From the implementation of _commit:

if ( !FlushFileBuffers((HANDLE)_get_osfhandle(filedes)) ) {
        retval = GetLastError();
}

Answer 3

I'm still a little unclear on what the question is. You start out by talking about managing the lifetime of a temporary file and then jump to wrapping an entire file i/o interface. Are you asking about how to manage a temporary file without the performance penalty of wrapping all the file I/O? Or are you interested in how the CRT functions can be faster than the WinAPI functions they are built on top of?

Several of the comparisons being made between the C run-time functions and the WinAPi functions are of the apples and oranges variety.

The C run-time functions buffer the I/O in library memory. There is another layer of buffering (and caching) in the OS.

fflush flushes the data from the library buffers to the OS. It may go directly to disk, or it may go to OS buffers for later writing. FlushFileBuffers gets data from the OS buffers onto the disk, which generally takes longer than moving data from the library buffers to the OS buffers.

Unaligned writes are expensive. The OS buffers make unaligned writes possible, but they don't really speed up the process. The library buffers may accept several writes before pushing data to the OS, effectively reducing the number of unaligned writes to the disk.

It's also possible (though this is just a guess) that the library routines are taking advantage of overlapped (asynchronous) I/O to the disk, where your straight-to-WinAPI implementation is all synchronous.

Answer 4

I might be crazy, but wouldn't it be easier to just write a replacement for tmpfile that uses fopen(temporaryname, "rwTD"), where you generate your own temporaryname?

At least then you don't have to worry about reimplementing <file.h>.