Process Synchronization by detecting file open (creation)

Question

There are two applications. The first application is remote to the machine in question (and I have NO ACESSS to it) and creates a large file via the network (LAN). I have no control over this process, nor do I know when it occurs. THIS IS WHAT I HAVE TO WORK WITH. I cannot add, change or alter this in any way.

The second application is written by me and processes this file when it is found. This app is scheduled to run every 5 mins.

A Situation could occur where the file is in the process of being written when my app attempts to process it resulting in an incomplete processing and/or other errors, so I need to detect if the file has been completely written BEFORE I start processing it.

I can use OpenFile() and request read/write locked access. An error would indicate that the file is being created.

I could possibly do something clever with file system watcher http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx

But I suspect there is an elegant way I have not thought of for windows (NT and later)

Answer 1

A completely hackish method would be to installed a API hook on the remote machine to trap the file close which would then launch your program or send a notification that its done.

Answer 2

The standard solution is for the writer to write to a temporary file and then rename the file when it's done.

That way the reader (your app) will only see the file when it is complete.

Answer 3

Back in the old days I had to write software which communicated using files on the network (between some *NIX application and a Windows application through a Novell fileserver) . To do this reliably we always convinced the provider of the data files to create an extra handshake file (just an empty file) when it completed writing the data. Our application would poll for the handshake file and if the handshake file existed, we read the data file. When our application finished reading the data it would delete the handshake file. The handshake file was monitored by the provider which did not touch the data file as long as the handshake file existed.

However, most of the time the provider of the data was willing to modify their application. I think the only way to do this reliably it to get a signal from the provider.

Answer 4

I think your system watcher solution via FileSystemWatcher or ReadDirectoryChanges is the best you can get. You can check out this tutorial in CodeProject too. How much more elegant could it get anyways?

Answer 5

Another answer but different: I suddenly recalled another project which did something similar. My application just used OpenFile() with OF_SHARE_EXCLUSIVE (I think) until it successfully opens the file. In this case this works fine because my application was accessing a file written by a remote application to a share on the same machine as my application. I never tested it with files on a file server but it can work. However file locking is not always reliable when used with remote files.

Answer 6

As many others already pointed out the only sane way would be an extra handshake. As this seems not to be possible I would do the following.

• Write your application to run permanently (like a daemon).
• Check for the file and if you find it
  • monitor it's size until it does no longer grow for a specified time (e.g. 30 seconds)
  • rename the file after you decided it's done transferring
  • process the renamed file