Java: Watching a directory to move large files

Question

I have been writing a program that watches a directory and when files are created in it, it changes the name and moves them to a new directory. In my first implementation I used Java's Watch Service API which worked fine when I was testing 1kb files. The problem that came up is that in reality the files getting created are anywhere from 50-300mb. When this happened the watcher API would find the file right away but could not move it because it was still being written. I tried putting the watcher in a loop (which generated exceptions until the file could be moved) but this seemed pretty inefficient.

Since that didn't work, I tried up using a timer that checks the folder every 10s and then moves files when it can. This is the method I ended up going for.

Question: Is there anyway to signal when a file is done being written without doing an exception check or continually comparing the size? I like the idea of using the Watcher API just once for each file instead of continually checking with a timer (and running into exceptions).

All responses are greatly appreciated!

nt

Answer 1

Write another file as an indication that the original file is completed. I.g 'fileorg.dat' is growing if done create a file 'fileorg.done' and check only for the 'fileorg.done'.

With clever naming conventions you should not have problems.

Answer 2

Two solutions:

The first is a slight variation of the answer by stacker:

Use a unique prefix for incomplete files. Something like myhugefile.zip.inc instead of myhugefile.zip. Rename the files when upload / creation is finished. Exclude .inc files from the watch.

The second is to use a different folder on the same drive to create / upload / write the files and move them to the watched folder once they are ready. Moving should be an atomic action if they are on the same drive (file system dependent, I guess).

Either way, the clients that create the files will have to do some extra work.

Answer 3

I speculate that java.io.File.canWrite() will tell you when a file has been done writing.