pyinotify bug with reading file on creation?

Question

Hi, this has been driving me crazy for the past 12 hours... I'm fairly new to Python, so pardon my ignorance. :)

I have a module that I want to parse a file everytime a new file is created in a certain directory. For this, I'm trying to use pyinotify to setup a directory to watch for IN_CREATE kernel events, and fire the parse() method.

Here is the module:

from pyinotify import WatchManager,
    ThreadedNotifier, ProcessEvent, IN_CREATE

class Watcher(ProcessEvent):

    watchdir = '/tmp/watch'

    def __init__(self):
        ProcessEvent.__init__(self)
        wm = WatchManager()
        self.notifier = ThreadedNotifier(wm, self)
        wdd = wm.add_watch(self.watchdir, IN_CREATE)
        self.notifier.start()

    def process_IN_CREATE(self, event):
        pfile = self._parse(event.pathname)
        print(pfile)

    def _parse(self, filename):
        f = open(filename)
        file = [line.strip() for line in f.readlines()]
        f.close()
        return file

if __name__ == '__main__':
    Watcher()

The problem is that the list returned by _parse is empty when triggered by a new file creation event, like so (the file is created in another window while watcher.py is running):

$ python watcher.py
[]

...but strangely enough, it works from an interpreter session when called directly.

>>> import watcher
>>> w = watcher.Watcher()
>>> w._parse('/tmp/watch/sample')
['This is a sample file', 'Another line', 'And another...']

Why is this happening? The farthest I've come debugging this thing is to know that something is making pyinotify not read the file correctly. But... why?

Thanks for any suggestions. I'm really losing my temper with this issue. :(

Answer 1

may be you want to wait till file is closed?

Answer 2

As @SilentGhost mentioned, you may be reading the file before any content has been added to file (i.e. you are getting notified of the file creation not file writes).

Update: The loop.py example with pynotify tarball will dump the sequence of inotify events to the screen. To determine which event you need to trigger on, launch loop.py to monitor /tmp and then perform the file manipulation you want to track.

Answer 3

Here's some code that works for me, with a 2.6.18 kernel, Python 2.4.3, and pyinotify 0.7.1 -- you may be using different versions of some of these, but it's important to make sure we're talking about the same versions, I think...:

#!/usr/bin/python2.4

import os.path
from pyinotify import pyinotify

class Watcher(pyinotify.ProcessEvent):

    watchdir = '/tmp/watch'

    def __init__(self):
        pyinotify.ProcessEvent.__init__(self)
        wm = pyinotify.WatchManager()
        self.notifier = pyinotify.ThreadedNotifier(wm, self)
        wdd = wm.add_watch(self.watchdir, pyinotify.EventsCodes.IN_CREATE)
        print "Watching", self.watchdir
        self.notifier.start()

    def process_IN_CREATE(self, event):
        print "Seen:", event
        pathname = os.path.join(event.path, event.name)
        pfile = self._parse(pathname)
        print(pfile)

    def _parse(self, filename):
        f = open(filename)
        file = [line.strip() for line in f]
        f.close()
        return file

if __name__ == '__main__':
      Watcher()

when this is running in a terminal window, and in another terminal window I do

echo "ciao" >/tmp/watch/c3

this program's output is:

Watching /tmp/watch
Seen: event_name: IN_CREATE   is_dir: False   mask: 256   name: c3   path: /tmp/watch   wd: 1   
['ciao']

as expected. So can you please try this script (fixing the Python version in the hashbang if needed, of course) and tell us the exact releases of Linux kernel, pyinotify, and Python that you are using, and what do you observe in these exact circunstances? Quite possibly with more detailed info we may identify which bug or anomaly is giving you problems, exactly. Thanks!

Answer 4

Thanks for your help, guys. I think I solved the problem using the IN_CLOSE _CREATE event instead. I'm not sure what was happening before that made it not work. :-S

@Alex: Thanks, I tried your script, but I'm using newer versions: Python 2.6.1, pyinotify 0.8.6 and Linux 2.6.28, so it didn't work for me. :(

It was definitely a matter of trying to parse the file before it was written, so kudos to SilentGhost and DanM for figuring it out. Much appreciated.

Sorry for replying so late, I see this is a fast moving discussion board. I'll try to keep up next time.

Cheers!