FileSystemWatcher and windows 7

Question

I am writing a tool that monitors a network directory and is running off of a Windows Server 2008 machine, the OnChanged event for the FileSystemWatcher is being fired correctly from files placed on the network drive by any computer that is not using Windows 7, for some reason if the amount of files copied is more than 19 on a windows 7 computer (at once) then no events are fired although it works if files are done individually. Is there a workaround for this or is that just how the Windows 7 kernel behaves with FSW events?

Answer 1

From MSDN:

The Windows operating system notifies your component of file changes in a buffer created by the FileSystemWatcher. If there are many changes in a short time, the buffer can overflow. This causes the component to lose track of changes in the directory, and it will only provide blanket notification. Increasing the size of the buffer with the InternalBufferSize property is expensive, as it comes from non-paged memory that cannot be swapped out to disk, so keep the buffer as small yet large enough to not miss any file change events. To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties so you can filter out unwanted change notifications.

If increasing the buffer size is not sufficient and you cannot control how many files are triggering events at a time you would have to add additional polling.

See also this related question:

FileSystemWatcher does not work properly when many files are added to the directory at the same time…

Update:

It might be tempting to simply increase the buffer size but this should be done with care. In fact, there is a 64k limitation when it comes to network access. The FileSystemWatcher class is using the Windows API function ReadDirectoryChangesW underneath which has this limit:

ReadDirectoryChangesW fails with ERROR_INVALID_PARAMETER when the buffer length is greater than 64 KB and the application is monitoring a directory over the network. This is due to a packet size limitation with the underlying file sharing protocols.

If you want to get a deeper understanding on the cost of modifying the buffer size you should have a look at the post of Walter Wang of Microsoft here:

FileSystemWatcher across the network (full post quoted below)

I'm sorry that the documentation of FileSystemWatcher.InternalBufferSize didn't state very clear about the buffer size when monitoring network path. It's recommended not exceeds 64K when monitoring network path.

FileSystemWatcher is basically a .Net wrapper for the Win32 ReadDirectoryChangesW API. To use ReadDirectoryChangesW, you create and specify a buffer that the OS will populate with the changes. However, what is not mentioned in the ReadDirectoryChangesW documentation (but is hinted in the FileSystemWatcher docs) is that the file system creates an internal kernel buffer to store the change information temporarily until it has the chance to update the user buffer. The size of the kernel buffer that is created is the same size that is specified in ReadDirectoryChangesW and is created in non-paged pooled memory. Every time a FileSystemWatcher / ReadDirectoryChangesW is created / called, a new kernel buffer is also created.

The kernel memory pools (paged and non-paged) are set aside in the system address space for device drivers and other kernel components to use. They grow and shrink dynamically as necessary. The current size of the pools can be easily seen by going to the Performance tab of the Task Manager. The pools will grow dynamically until they hit a maximum value which is calculated at boot time and depends on available system resources (mostly RAM). You do not want to hit this maximum value or else various system services and drivers will start failing. However, this calculated maximum value is not easily available. To determine the maximum pool sizes, you need to use a kernel debugger. If you are interested in further information about the system memory pools, I recommend that you look at Chapter 7 in the MSPress book Inside Windows 2000 by Solomon and Russinovich.

With this in mind, there is no recommendation on what size buffers you can use. The current and maximum size of the system pools are going to be varied from client to client. However, you probably should not go over 64k for each FileSystemWatcher / ReadDirectoryChangesW buffer. This stems from the fact that there is a 64k limitation with network access as documented in ReadDirectoryChangesW. But in the end you are going to have to test the application on a variety of expected target systems so that you can tune your buffer.

There is overhead associated with .Net applications and I imagine that a Win32 ReadDirectoryChangesW program might be able to achieve better performance with the same buffer size. However, with very fast and numerous file changes, buffer overruns will be inevitable and the developer is going to have to handle the case when an overrun occurs such as manually enumerating the directory to detect the changes.

In conclusion, FileSystemWatcher and ReadDirectoryChangesW are a lightweight file change detection mechanism that is going to have its limitations. Change Journals is another mechanism which we would consider a medium-weight solution, but would still have limitations:

http://msdn.microsoft.com/en-us/library/aa363798%28VS.85%29.aspx

Heavy-weight solutions would be to write a dedicated file system filter driver that sits in the file system stack and monitors file system changes. Of course this would be the most complex approach. Most virus scanners, backup software, and file system monitoring utilities such as filemon (www.sysinternals.com) implement a filter driver.

I hope above explanation helps you to understand the root cause of the issue you're experiencing. Please reply to let us know whether or not you need further information. Thank you.

Answer 2

After many many attempts using the FileSystemWatcher I have given up on it. It won't fire events correctly, at the wrong time, the incorrect type. Honestly I think it's one of the worst classes in the .net framework. I have always ended up writing my own class that takes a System.Timer and after x milliseconds elapsed it will check directories, files manually. Yes it takes more work and yes it can be a slight PITA but once you've written it you can use it wherever you want. I wish the FileSystemWatcher worked as advertised but I've never found it to do so.

Answer 3

Most of the times when a FileSystemWatcher event fires, I ignore the files that are being passed by the object because this list may be incomplete.

I simply crawl through the directory that the FileSystemWatcher object is monitoring and perform a manual scan to find the files that might not be in the FileSystemWatcher buffer. It's not really elegant, but it makes sure you won't miss a file.

Answer 4

This is one really unreliable class, when files are above a certain threshold I've had events firing about 7 times on the initial copy and then when the file transfer dialog is done I get another 7 events, this issue appears across all OS'es that support the FSW. Although Windows 7 (haven't tried vista yet) still only accepts files one by one or 19 at a time while if I drop files from an XP machine into the network drive I can read thousands of files at once with no issues whatsoever. This may just be a change to ReadDirectoryChangesW from XP to 7. Past 19 files I could not get ANY events to fire so I guess that will become a "feature" of my tool for now. If anyone has any other info feel free to contribute.