Using pyinotify to watch for file creation, but waiting for it to be completely written to disk

Question

I'm using pyinotify to watch a folder for when files are created in it. And when certain files are created I want to move them. The problem is that as soon as the file is created (obviously), my program tries to move it, even before it's completely written to disk.

Is there a way to make pyinotify wait until a file is completely written to disk before notifying me that it's been created? Or is there any easy way to, after I'm notified, make python wait to move it until it's done being written?

Answer 1

It is quite difficult to tell at this level if a file is being written to. What you can do is test to see if a file is opened by some other process.

1) From the various flags that are used while opening a file, O_EXLOCK flag might be of help. If the O_EXLOCK flag is set, the file descriptor has an exclusive lock on the file. So my understanding is if you can do os.open() with O_EXLOCK flag, it's not open by other process. This should work on all posix compatible OS but I have not tested it. If the file, is open then you could close, wait and retry again.

2) You can also try os.stat and see changing time stamp and try to safely interpret the information. Though this is not fool proof.

3) On unix systems, you can try "lsof"

4) The following page describes use of symlinks from /proc/PID/fd to test for open files

• http://www.gossamer-threads.com/lists/python/python/639874

[Edit : Links updated]

• https://launchpad.net/ubuntu/%2Bsource/lsof

Answer 2

Have pyinotify react to IN_CLOSE_WRITE events:

wm.add_watch(watched_dir, pyinotify.IN_CLOSE_WRITE, proc_fun=MyProcessEvent())

This is from man 5 incrontab, but it applies equally well to pyinotify:

   IN_ACCESS           File was accessed (read) (*)
   IN_ATTRIB           Metadata changed (permissions, timestamps, extended attributes, etc.) (*)
   IN_CLOSE_WRITE      File opened for writing was closed (*)
   IN_CLOSE_NOWRITE    File not opened for writing was closed (*)
   IN_CREATE           File/directory created in watched directory (*)
   IN_DELETE           File/directory deleted from watched directory (*)
   IN_DELETE_SELF           Watched file/directory was itself deleted
   IN_MODIFY           File was modified (*)
   IN_MOVE_SELF        Watched file/directory was itself moved
   IN_MOVED_FROM       File moved out of watched directory (*)
   IN_MOVED_TO         File moved into watched directory (*)
   IN_OPEN             File was opened (*)

Answer 3

If you have control of the writing process, you could call the file "foo.part" while it is being written to and rename it to "foo" when it has been closed.