"Reliable" file work with .NET

Question

Hi,

An application needs a counter, which value to be stored in a text file. Sometimes it happens on very short intervals.

This test code rewrites a text file very often (f.e. every 100 milliseconds):

int counter = 0;

while (true) {

WriteToFile(counter);
counter++;
Thread.Sleep(100);

}

private void WriteToFile(int counter) {

byte[] buffer = Encoding.ASCII.GetBytes(counter.ToString());
using (FileStream createFile = new FileStream("Counter.txt", FileMode.Create))
{
    createFile.Write(buffer, 0, buffer.Length);
    createFile.Close();
}

}

It works basically fine except in one of our "reliable tests" - stopping the computer electricity supply while the application is running. The bad surprise was - in the file wasn't any text (should be a number) but only one or two space characters. The missing last value it tries to write is understandable but making the file not consistent is really bad...

Trying the use of:

createFile.Flush(); createFile.Close(); createFile.Dispose();

GC.Collect(); GC.WaitForPendingFinalizers();

doesn't help.

Any help will be appreciated.

Answer 1

I think the easiest thing to do would be to use the transactional filesystem in Windows Vista/2008:

http://www.michaelckennedy.net/blog/2007/12/07/SystemTransactionsAndWindowsVistaNTFSUpdated.aspx

http://msdn2.microsoft.com/magazine/2fc4ae05-f7b8-49d2-8630-f24bc9dfc2e6

That is, if you're one of the few who run it...

Answer 2

Various options spring to mind:-

1) keep a temporary copy of the current file, e.g. rename old file, write new file, delete old file (or write new file with temporary name, delete old file, rename new file), but perhaps your are writing out the file too often for this kind of thing

2) write multiple files, with a periodic clean up, e.g. 0001.txt, 0002.txt, 0003.txt, ... etc. and delete all but last file every 10 seconds.

3) write to the same file multiple times, perhaps creating a new file periodically, say every 10 minutes, again with periodic clean up.

4) Use a third party logging platform such as log4net which probably safeguards against things like power outages, and can log to various targets, e.g. file, database, IRC

Answer 3

Your file is missing values because the power interruption closed it before Windows had a chance to flush it out to disk. You can call Flush, Close, Dispose, etc. all you want but they can't operate if a power failure interrupts the call. At best, they'll reduce the window of vulnerability.

Adam Ralph's answer is, I think, the most appropriate one for running on any system. I voted for him but wanted to add my $0.02 regarding the failure of your various system calls to correct this problem.

Answer 4

turn off "optimise for performance" in the filesystem partition. You're writing the file, but Windows is not bothering to write the data to the physical disk, instead it keeps it in memory intending to write it eventually, ie. when there's some idle time.

Obviously you're not giving it any idle time :)

So, turn off that flag and make Windows write the contents to disk every time. This is like the same flag you use for flash drives to ensure the data is actually written before you pull the thing out the USB port. You can force Windows to write the data using the FlushFileBuffers API call.

Or you could use the FILE_FLAG_WRITE_THROUGH option.

Obviously, make sure your hardware is writing the data too - if you have a RAID card, it will be caching data writes itself, you'll need a battery backing it up to ensure the data is physically written to the drive.

I think you won't be happy with the performance though. Your best bet is then to write to disk using a battery-backed raid card.

Answer 5

Thank you all for the answers.

Listening to them the solution for now has become:

using (FileStream createFile = new FileStream("Counter.txt", FileMode.Create, FileAccess.Write, FileShare.None, 8, FileOptions.WriteThrough)) {}

All parameters are included because of the last one - FileOptions.WriteThrough. Its purpose is avoiding the use of every kind of cash, buffers, memory and so on.

As this solution doesn't offer 100% guarantees (because of hardware reasons :) ) a back-up is kept. In fact the every write is done twice - in the original and in the back-up file.