File access error with FileSystemWatcher when multiple files are added to a directory

Question

I am running into an issue with a FileSystemWatcher when multiple files are placed into the watched directory. I want to parse the file as soon as it is placed in the directory. Typically, the first file parses fine, but adding a second file to the directory causes an access issue. Occasionally, the first file doesn't even parse. There is only one application running and watching this directory. Eventually, this process will be running on multiple machines and they will be watching a shared directory but only one server can parse each file as the data is imported into a database and there are no primary keys.

Here is the FileSystemWatcher code:

public void Run() {
  FileSystemWatcher watcher = new FileSystemWatcher("C:\\temp");
  watcher.NotifyFilter = NotifyFilters.FileName;
  watcher.Filter = "*.txt";

  watcher.Created += new FileSystemEventHandler(OnChanged);

  watcher.EnableRaisingEvents = true;
  System.Threading.Thread.Sleep(System.Threading.Timeout.Infinite);
}

Then the method that parses the file:

private void OnChanged(object source, FileSystemEventArgs e) {
  string line = null;

  try {
    using (FileStream fs = new FileStream(e.FullPath, FileMode.Open, FileAccess.Read, FileShare.None)) {
      using (StreamReader sr = new StreamReader(fs)) {
        while (sr.EndOfStream == false) {
          line = sr.ReadLine();
          //parse the line and insert into the database
        }//end while
      }//end StreamReader using
    }//end FileStream using
  }//end try
  catch (IOException ioe) {
    Console.WriteLine("OnChanged: Caught Exception reading file [{0}]", ioe.ToString());
  }

When moving the second file, it is catching

System.IO.IOException: The process cannot access the file 'C:\Temp\TestFile.txt' because it is being used by another process.

I would expect to see this error if it was running on multiple machines, but it is only running on one server for now. There shouldn't be another process using this file - I have them created and copy them into the directory when the application is running.

Is this the proper way to set up the FileSystemWatcher? How can I see what has the lock on this file? Why doesn't it parse both files - do I have to close the FileStream? I want to keep the FileShare.None option because I only want one server to parse the file - the server that gets to the file first parses it.

Answer 1

A typical problem of this approach is that the file is still being copied while the event is triggered. Obviously, you will get an exception because the file is locked during copying. An exception is especially likely on large files.

As a workaround you could first copy the file and then rename it and listen to the renaming event.

Or another option would be to have a while loop checking whether the file can be opened with write access. If it can you will know that copying has been completed. C# code could look like this (in a production system you might want to have a maximum number of retries or timeout instead of a while(true)):

/// <summary>
/// Waits until a file can be opened with write permission
/// </summary>
public static void WaitReady(string fileName)
{
    while (true)
    {
        try
        {
            using (Stream stream = System.IO.File.Open(fileName, FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite))
            {
                if (stream != null)
                {
                    System.Diagnostics.Trace.WriteLine(string.Format("Output file {0} ready.", fileName));
                    break;
                }
            }
        }
        catch (FileNotFoundException ex)
        {
            System.Diagnostics.Trace.WriteLine(string.Format("Output file {0} not yet ready ({1})", fileName, ex.Message));
        }
        catch (IOException ex)
        {
            System.Diagnostics.Trace.WriteLine(string.Format("Output file {0} not yet ready ({1})", fileName, ex.Message));
        }
        catch (UnauthorizedAccessException ex)
        {
            System.Diagnostics.Trace.WriteLine(string.Format("Output file {0} not yet ready ({1})", fileName, ex.Message));
        }
        Thread.Sleep(500);
    }
}

Yet another approach would be to place a small trigger file in the folder after copying is completed. Your FileSystemWatcher would listen to the trigger file only.

Answer 2

I had the same problem within DFS. My resolution was achived by adding two empty lines to each file. Then my code waits for two empty lines in file. Then I have certainty to read whole data from file.

Answer 3

When you open the file in your OnChanged method, you're specifying FileShare.None, which according to the documentation, will cause any other attempts to open the file to fail while you've got it open. Since all you (and your watcher) are doing is reading, try using FileShare.Read instead.