What is the most efficient method to write data to the file?

Question

I need to write a large amount of data in my application. Data arrive periodically and pushed into the queue.

//producer
queue_.push( new_data );// new_data is a 64kb memory range
PostThreadMessage(worker_thread_id_, MyMsg, 0, 0);//notify the worker thread

after that, i need to write this data to the file. I have another thread, that get data from the queue and writes it.

//consumer
while(GetMessage(msg, 0, 0))
{
   data = queue_.pop();
   WriteFile(file_handle_, data.begin(), data.size(), &io_bytes, NULL);
}

This is simple, but not efficient. Another oportunity is to use IO completion ports.

//producer
//file must be associated with io completion port
WriteFile( file_handle_
         , new_data.begin()
         , new_data.size()
         , &io_bytes
         , new my_overlaped(new_data) );

consumer is just delete unused buffers

my_completion_key* ck = 0;
my_overlapped* op = 0;
DWORD res = GetQueuedIOCompletionStatus(completion_port_, &io_bytes, &ck, &op);
//process errors
delete op->data;

What is the preferred way, to perform large amount of file i/o operation on windows?

Answer 1

Hard to tell because it is not clear what kind of problem you are experiencing with your current approach. Going async in a dedicated "writer" thread shouldn't result in considerable performance improvement.

If you have small but frequent writes (so that frequent WriteFile's cost becomes a bottleneck) I would suggest implementing a kind of lazy writes - collect data and flush it to file when some volume threshold is reached.