rewriting the same file in rapid succession?

Question

I am working on an app that will keep a running index of work in accomplished.

I could write once at the end of a work session, but I don't want to risk losing data if something blows up. Therefore, I rewrite to disk (XML) every time a new entry or a correction is made by the user.

private void WriteIndexFile()
{
    XmlDocument IndexDoc
    // Build document here

    XmlTextWriter tw = new XmlTextWriter(_filePath, Encoding.UTF8);
    tw.Formatting = Formatting.Indented;
    IndexDoc.Save(tw);
}

It is possible for the writes to be triggered in rapid succession. If this happens, it tries to open the file for writing before the prior write is complete. (While it would not be normal, I suppose it is possible that the file gets opened for use by another program.)

How can I check if the file can be re-written?

Edit for clarification: This is part of an automated lab data collection system. The users will click a button to capture data (saved in separate files), and identify the sub-task the the data package is for. Typically, it will be 3-10 minutes between clicks.

If they make an error, they need to be able to go back and correct it, so it's not an append-only usage.

Finally, the files will be read by other automated tools and manually by humans. (XML/XSLT)

The size will be limited as each work session (worker shift or less) will have a new index file generated.

Further question: As the overwhelming consensus is to not use XML and write in an append-only mode, how would I solve the requirement of going back and correcting earlier entries?

I am considering having a "dirty" flag, and save a few minutes after the flag is set and upon closing the work session. If multiple edits happen in that time, only one write will occur - no more rapid user - also have a retry/cancel dialog if the save fails. Thoughts?

Answer 1

Use an inter-process Mutex

Answer 2

I would suggest you maintain a separate "Writer" thread to which you queue up the write requests using a synchronized queue in memory.

So any write you need, you write to the queue and continue. The Writer threads sole job is to wait for things to appear in the queue and write it out to the file.

Answer 3

If all you want is a simple log of all operations, XML may be the wrong choice here as it is difficult to append to an XML document without rewriting the whole file, which will become slower and slower as the file grows.

I'd suggest instead File.AppendText or even better: keeping the file open for the duration of the aplication's life time and using WriteLine.

(Oh, and as others have pointed out, you need to lock to ensure that only one thread writes to the file at a time. This is still true even with this solution.)

There are also logging frameworks that already solve this problem, such as log4net. Have you considered using an existing logging framework instead of rolling your own?

Answer 4

XML is a poor choice in your case because new content has to be inserted before the closing tag. Use Text istead and simply open the file for append and write the new content at the end of the file, see How to: Open and Append to a Log File.

You can also look into a simple logging framework like log4net and use that instead of handling the low level file stuff urself.

Answer 5

I have a logger that uses System.Collections.Queue. Basically it waits until something is queued then trys to write it. While writing items, which could be slow, more items could be added to the queue.

This will also help in just grouping messages rather than trying to keep up. It is running on a separate thread.

private AutoResetEvent ResetEvent { get; set; }

LogMessage(string fullMessage)
{
    this.logQueue.Enqueue(fullMessage);

    // Trigger the Reset Event to send the 
    this.ResetEvent.Set();
}

private void ProcessQueueMessages()
{
    while (this.Running)
    {
        // This will process all the items in the queue.
        while (this.logQueue.Count > 0)
        {
            // This method will just log the top item on the queue
            this.LogQueueItem();
        }

        // Once the queue is empty will wait for a 
            // another message to queueed before running again.  
        // Rather than sleeping and checking if the queue is full, 
            // saves from doing a System.Threading.Thread.Sleep(1000); stuff
        this.ResetEvent.WaitOne();
    }
}

I handle write failures but not dequeueing until it wrote to the file with no errors. Then I just keep attempting until it finally can write. This has saved me because somebody removed permissions from one of our apps during it process. Permission was given back with out shutting down our app, and we didn't lose a single log statement.

Answer 6

Consider using a flat text file. I have a process that I wrote that uses an XML log... it was a poor choice. You can't just write out the state as you run without having to constantly rewrite the file to make sure the tags are correct. If it was flat entries written to a file you could have an automatic timeline that could give you details of what happened without trying to figure out if it was the XML writer/tag set that blew up and you don't have to worry about your logs bloating out as much.

Answer 7

I agree with others suggesting you avoid XML. Also, I would suggest you have one component (a "monitor") that is responsible for all access to the file. That component will have the job of handling multiple simultaneous requests and making the disk writes happen one after another.