System.IO.FileSystemWatcher to monitor a network-server folder - Performance considerations

Question

I want to watch a folder tree on a network server for changes. The files all have a specific extension. There are about 200 folders in the tree and about 1200 files with the extension I am watching.

I can't write a service to run on the server (off-limits!) so the solution has to be local to the client. Timeliness is not particularly important. I can live with a minute or more delay in notifications. I am watching for Create, Delete, Rename and Changes.

Would using the .NET System.IO.fileSystemWatcher create much of a load on the server?

How about 10 separate watchers to cut down the number of folders/files being watched? (down to 200 from 700 folders, 1200 from 5500 files in total) More network traffic instead of less? My thoughts are a reshuffle on the server to put the watched files under 1 tree. I may not always have this option hence the team of watchers.

I suppose the other solution is a periodic check if the FSW creates an undue load on the server, or if it doesn't work for a whole bunch of SysAdmin type reasons.

Is there a better way to do this?

Answer 1

I wouldn't think there's any sort of active state or communication between the computer with the FSW and the computer whose location is being monitored. In other words, the FSW isn't pinging the networked OS to check on the file.

One would imagine that a message or event is only raised/sent to the networked FSW when a change occurs.

But this is all just speculation. :)

Answer 2

If you are running Windows Server 2003 R2 or Windows Server 2008, there is built in File System Watching.

Answer 3

The MSDN documentation indicates that you can use the FileSystemWatcher component to watch for filesystem changes on a network drive.

It also indicates that the watcher component listens for file system change notifications rather than periodically interrogating the target drive for changes.

Based on that the amount of network traffic depends entirely on how much you expect the contents of that network drive to change. The FSW component will not add to the level of network traffic.

Answer 4

It probably hooks to WMI.

Answer 5

From a server load point of view, using the IO.FileSystemWatcher for remote change notifications in the scenario you describe is probably the most efficient method possible. It uses the FindFirstChangeNotification and ReadDirectoryChangesW Win32 API functions internally, which in turn communicate with the network redirector in an optimized way (assuming standard Windows networking: if a third-party redirector is used, and it doesn't support the required functionality, things won't work at all). The .NET wrapper also uses async I/O and everything, further assuring maximum efficiency.

The only problem with this solution is that it's not very reliable. Other than having to deal with network connections going away temporarily (which isn't too much of a problem, since IO.FileSystemWatcher will fire an error event in this case which you can handle), the underlying mechanism has certain fundamental limitations. From the MSDN documentation for the Win32 API functions:

• ReadDirectoryChangesW fails with ERROR_INVALID_PARAMETER when the buffer length is greater than 64 KB and the application is monitoring a directory over the network. This is due to a packet size limitation with the underlying file sharing protocols

• Notifications may not be returned when calling FindFirstChangeNotification for a remote file system

In other words: under high load (when you would need a large buffer) or, worse, under random unspecified circumstances, you may not get the notifications you expect. This is even an issue with local file system watchers, but it's much more of a problem over the network. Another question here on SO details the inherent reliability problems with the API in a bit more detail.

When using file system watchers, your application should be able to deal with these limitations. For example:

• If the files you're looking for have sequence numbers, store the last sequence number you got notified about, so you can look for 'gaps' on future notifications and process the files for which you didn't get notified;

• On receiving a notification, always do a full directory scan. This may sound really bad, but since the scan is event-driven, it's still much more efficient than dumb polling. Also, as long as you keep the total number of files in a single directory, as well as the number of directories to scan, under a thousand or so, the impact of this operation on performance should be pretty minimal anyway.

Setting up multiple listeners is something you should avoid as much as possible: if anything, this will make things even less reliable...

Anyway, if you absolutely have to use file system watchers, things can work OK as long as you're aware of the limitations, and don't expect 1:1 notification for every file modified/created.

So, if you have other options (essentially, having the process writing the files notify you in a non-file system based way: any regular RPC method will be an improvement...), those are definitely worth looking into from a reliability point of view.

Answer 6

I've used the file system watchers from C# a number of times. The first time I used them, I had problems with them stopping working, mainly due to the fact that I was processing the changes in the thread that reported the change.

Now however, I just push the change onto a queue and process the queue on another thread. This seems to solve the problem I originally had. For your problem, you could have multiple watchers pushing onto the same queue.

However, I haven't used this with your sort of scale of problem.

Answer 7

In my experience, an FSW does not create high network traffic. However, if there is a performance problem, your approach of using several watchers and breaking it down to fewer folders being watched sounds reasonable.

I had some big problems with FSW on network drives, though: Deleting a file always threw the error event, never the deleted event. I did not find a solution, so I now avoid using FSW if there is a way around it...