I need to setup an application that watches for files being created in a folder (locally or on a network drive) and I was wondering if anyone has any thoughts on whether the FileSystemWatcher or polling on a timer would be the best option. I have used both methods in the past, but not extensively. Have you run into any issues (performance, reliability... etc) with either method? I know there isn't a "right way" to do this, I'm just looking opinions.
I wouldn't consider using polling "reinventing the wheel". As noted by some of the other answers, the FileSystemWatcher can be unreliable in certain situations. This is what I was afraid of, and I really don't want to code a FileSystemWatcher method that also polls.
Jon Tackabury
2008-10-27 14:28:19
+26
A:
I have seen the file system watcher fail in production and test environments. I now consider it a convenience, but I do not consider it reliable. My pattern has been to watch for changes with the files system watcher, but poll occasionally to catch missing file changes.
Edit: If you have a UI, you can also give your user the ability to "refresh" for changes instead of polling. I would combine this with a file system watcher.
Jason Jackson
2008-10-27 14:12:51
I've seen if fall down, too. The solution we've used is to wrap our own class around, where the wrapper class ALSO uses a timer to check on occasion that the watcher is still going.
Joel Coehoorn
2008-10-27 14:16:11
We do something similar - once we've processed the file passed into the FileCreated event, we do a manual check for any other new files before returning. This seem to mitigate any problems occurring with lots of files arriving at once.
John Sibly
2008-10-27 14:19:03
Thanks for the tips - I have heard that the FileSystemWatcher can be unreliable at times, it makes me a bit hesitant to depend on it in a production application.
Jon Tackabury
2008-10-27 14:29:39
I agree with writing an abstraction layer. The client code should receive the same event whether a polling created it, or the FileSystemWatcher did. Same with the manual "Refresh" command.
Jason Jackson
2008-10-27 14:30:09
Cristian Ciupitu
2008-10-27 14:47:09
I believe we tested it in XP and Server 2003 on a local directory and a file share, and had XP machines in the field. We had problems with both local dir and file share. One of the probable causes we came up with was the copy/creation of a lot of files in a short amount of time in the directory.
Jason Jackson
2008-10-27 17:26:14
+9
A:
Also note that file system watcher is not reliable on file shares. Particularly if the file share is hosted on a non-windows server. FSW should not be used for anything critical. Or should be used with an occasional poll to verify that it hasn't missed anything.
chilltemp
2008-10-27 14:17:33
+12
A:
The filesystemwatcher may also miss changes during busy times, if the number of queued changes overflows the buffer provided. This is not a limitation of the .net class per se, but of the underlying Win32 infrastructure. In our experience, the best way to minimize this problem is to dequeue the notifications as quickly as possible and deal with them on another thread.
As mentioned by ChillTemp above, the watcher may not work on non-Windows shares. For example, it will not work at all on mounted Novell drives.
I agree that a good compromise is to do an occasional poll to pick up any missed changes.
Brent Rockwood
2008-10-27 14:23:41
A:
I have run into trouble using FileSystemWatcher on network shares. If you're in a pure Windows environment, it might not be an issue, but I was watching an NFS share and since NFS is stateless, there was never a notification when the file I was watching changed.
Jon Norton
2008-10-27 14:24:53
+5
A:
Personally, I've used the FileSystemWatcher on a production system, and it has worked fine. In the past 6 months, it hasn't had a single hiccup running 24x7. It is monitoring a single local folder (which is shared). We have a relatively small number of file operations that is has to handle (10 events fired per day). It's not something I've ever had to worry about. I'd use it again if I had to remake the decision.
Jim
2008-10-27 14:32:53
A:
I had some big problems with FSW on network drives: Deleting a file always threw the error event, never the deleted event. I did not find a solution, so I now avoid the FSW and use polling.
Creation events on the other hand worked fine, so if you only need to watch for file creation, you can go for the FSW.
Also, I had no problems at all on local folders, no matter if shared or not.
Treb
2008-10-27 14:35:22
+7
A:
The biggest problem I have had is missing files when the buffer gets full. Easy as pie to fix--just increase the buffer. Remember that it contains the file names and events, so increase it to the expected amount of files (trial and error). It does use memory that cannot be paged out, so it could force other processes to page if memory gets low.
Here is the MSDN article on buffer : FileSystemWatcher..::.InternalBufferSize Property
Per MSDN:
Increasing buffer size is expensive, as it comes from non paged memory that cannot be swapped out to disk, so keep the buffer as small as possible. To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties to filter out unwanted change notifications.
We use 16MB due to a large batch expected at one time. Works fine and never misses a file.
We also read all the files before beginning to process even one...get the file names safely cached away (in our case, into a database table) then process them.
For file locking issues I spawn a process which waits around for the file to be unlocked waiting one second, then two, then four, et cetera. We never poll. This has been in production without error for about two years.