FileSystemWatcher for notification

Question

Hi, everyone. I'm creating a Windows application which use FileSystemWatcher. FileSystemWatcher watches some catalog for changes. And every time some file is added to this catalog FileSystemWatcher must add information about this file to an XML file. All works fine, but, when I add, for example, 100 files at same time (say that some application adds these files to the catalog) not every file's information appears in this xml file.

I want to use Queue. And use it to add items to this collection. And use a timer. The timer will add information to the XML from this collection. Is it good idea?

Can anyone advise me what to do?

Answer 1

FileSystemWatcher has an internal Buffer for changes. When there are a lot of quick changes, the buffer may not catch all Events.

Try increasing the InternalBufferSize to something higher.

You can set the buffer to 4 KB or larger, but it must not exceed 64 KB. For best performance, use a multiple of 4 KB on Intel-based computers.

The system notifies the component of file changes, and it stores those changes in a buffer the component creates and passes to the APIs. Each event can use up to 16 bytes of memory, not including the file name. If there are many changes in a short time, the buffer can overflow. This causes the component to lose track of changes in the directory, and it will only provide blanket notification. Increasing the size of the buffer has the following ramifications:

Increasing the buffer size can prevent missing file system change events. Note that an instance of the FileSystemWatcher class can raise an Error event when an event is missed or when the buffer size is exceeded, due to dependencies with the Windows operating system.

Increasing buffer size is expensive, as it comes from non paged memory that cannot be swapped out to disk, so keep the buffer as small as possible. To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties to filter out unwanted change notifications.

For diagnostics, maybe you should first subscribe to the Error Event to see if it's really a Buffer Overflow.

Also as said, set the NotifyFilter to the smallest required flags, which could be NotifyFilters.LastWrite if you only want to track changes.

Answer 2

If I understand correctly: You're watching a directory, and when many files get added at the same time you only see some of them with the FileSystemWatcher.

Have you tried the following: in the event handler for OnCreated, just go to the filesystem and get the complete contents of the directory, regardless of what the event tells you?

Answer 3

As Michael Stum writes in his answer, you could try increasing the buffer size (FileSystemWatcher.InternalBufferSize). Note however that you shouldn't set this value to too high a value. Also, IMHO this is possibly only a temporary fix, see what happens when you add yet more files to your folder at the same time.

I have read about some other things you can attempt if increasing the buffer size doesn't help:

• If you subscribe to the notification events of FileSystemWatcher, try to keep your event handler as short as possible; ie. make sure execution doesn't stay there long. If you have to do a fair amount of work per file notification, you could try to start a separate thread and do the processing there; your event handler could then get back to the caller very quickly (and the file notification would get removed from the buffer/queue sooner).

• Don't use any additional information provided by FileSystemWatcher apart from the basic notification that something has changed. Once you get any file change notification, wait a brief timespan until no more notifications arrive (ie. wait for the last notification out of 100 simultaneous file change notifications). Then list the contents of the directory manually and transfer the required information to your XML.

  This has one major drawback: Manually detecting whether, and which, files have been deleted, renamed, or created is not easy. Your program would have to keep the last directory listing against which the current listing can be compared to find out what's changed.

  The advantage of this is that you can be fairly sure that no changes will be dropped due to some FileSystemWatcher buffer overflow.

Answer 4

So i think that i must create a Windows application and a Windows Service. WinApp will only add information to EventLog and Windows Service will read information EventLog and write it to XML. I think it will the best way to do it. I'm waiting for good advice

Answer 5

There is a critical note in the MSDN docs that may help you in more reliably detecting changes:

Keep your event handling code as short as possible.

I suspect (but do not know for sure) that this is because filesystem events are raised in the main watcher thread, so any time you spend handling events creates a window where changes can go undetected. The way you have described your solution, it seems likely that you are doing I/O in the callback (the writes to your XML change log file), and that could definitely be too much work to do inline, by the criteria in the API docs. If you have lots of work to do for an event, hand off your events for processing in a separate thread so that you can get back to watching the filesystem asap.

One relatively easy way to do that would be to use ThreadPool.QueueUserWorkItem. This means that your change log will still not be 100% in sync with the state of the filesystem (due to latency introduced by use of separate threads and the queue) but it may be more accurate, and that seems to be your main concern. You would need to make sure that your WaitCallback invoked by the threadpool is thread-safe (e.g. writes to your change log do not happen concurrently without a lock() or similar), and be aware that there's no guarantee that change log entries will be written in the order they occurred (though it's doubtful to me whether FileSystemWatcher guarantees that anyway).

Also per the guidelines for the API - make sure you are filtering the possible set of events so you only get called back for the ones you absolutely must see:

To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties so you can filter out unwanted change notifications.