python automate ffmpeg conversion from upload directory

Question

I have a upload script done. But i need to figure out how to make a script that I can run as a daemon in python to handle the conversion part and moving the file thats converted to its final resting place. heres what I have so far for the directory watcher script:

 #!/usr/bin/python

import os
import pyinotify import WatchManager, Notifier, ThreadedNotifier, ProcessEvent, EventCodes
import sys, time, syslog, config
from os import system
from daemon import Daemon

class myLog(ProcessEvent):
 def process_IN_CREATE(self, event):
  syslog.syslog("creating: " + event.pathname)
 def process_IN_DELETE(self, event):
  syslog.syslog("deleting: " + event.pathname)
 def process_default(self, event):
  syslog.syslog("default: " + event.pathname)

class MyDaemon(Daemon):
 def loadConfig(self):
  """Load user configuration file"""
  self.config = {}
  self.parser = ConfigParser.ConfigParser()
  if not os.path.isfile(self.configfile):
   self.parser.write(open(self.configfile, 'w'))
  self.parser.readfp(open(self.configfile, 'r'))

  variables = { \
   'mplayer':  ['paths', self.findProgram("mplayer")], \
   'mencoder':  ['paths', self.findProgram("mencoder")], \
   'tcprobe':  ['paths', self.findProgram("tcprobe")], \
   'transcode':  ['paths', self.findProgram("transcode")], \
   'ogmmerge':  ['paths', self.findProgram("ogmmerge")], \
   'outputdir':  ['paths', os.path.expanduser("~")], \
   }

  for key in variables.keys():
   self.cautiousLoad(variables[key][0], key, variables[key][1])

 def cautiousLoad(self, section, var, default):
  """Load a configurable variable within an exception clause,
  in case variable is not in configuration file"""
  try:
   self.config[var] = int(self.parser.get(section, var))
  except:
   self.config[var] = default
   try:
    self.parser.set(section, var, default)
   except:
    self.parser.add_section(section)
    self.parser.set(section, var, default)
   self.parser.write(open(self.configfile, 'w'))


 def findProgram(self, program):
  """Looks for program in path, and returns full path if found"""
  for path in config.paths:
   if os.path.isfile(os.path.join(path, program)):
    return os.path.join(path, program)
  self.ui_configError(program)

 def run(self):
  syslog.openlog('mediaConvertor', syslog.LOG_PID,syslog.LOG_DAEMON)
  syslog.syslog('daemon started, entering loop')
  wm = WatchManager()
  mask = IN_DELETE | IN_CREATE
  notifier = ThreadedNotifier(wm, myLog())
  notifier.start()
  wdd = wm.add_watch(self.config['outputdir'], mask, rec=True)
  while True:
   time.sleep(1)
  wm.rm_watch(wdd.values())
  notifier.stop()
  syslog.syslog('exiting media convertor')
  syslog.closelog()

if __name__ == "__main__":
 daemon = MyDaemon('/tmp/mediaconvertor.pid')
 if len(sys.argv) == 2:
  if 'start' == sys.argv[1]:
   daemon.run()
  if 'stop' == sys.argv[1]:
   daemon.stop()
  if 'restart' == sys.argv[1]:
   daemon.restart()
  else:
   print "Unknown Command"
   sys.exit(2)
  sys.exit(0)
 else:
  print "Usage: %s start|stop|restart" % sys.argv[0]
  sys.exit(2)

not sure where to go from here.

Answer 1

I don't run on Linux and have never used the inotify capabilities you are using here. I'll describe how I would do things generically.

In the simplest case, you need to check if there's a new file in the upload directory and when there is one, start doing the conversion notification.

To check if there are new files you can do something like:

import os
import time

def watch_directory(dirname="."):
    old_files = set(os.listdir(dirname))
    while 1:
        time.sleep(1)
        new_files = set(os.listdir(dirname))
        diff = new_files - old_files
        if diff:
            print "New files", diff
        old_files = new_files

watch_directory()

You may be able to minimize some filesystem overhead by first stat'ing the directory to see if there are any changes.

def watch_directory(dirname="."):
    old_files = set(os.listdir(dirname))
    old_stat = os.stat(dirname)
    while 1:
        time.sleep(1)
        new_stat = os.stat(dirname)
        if new_stat == old_stat:
            continue
        new_files = set(os.listdir(dirname))
        diff = new_files - old_files
        if diff:
            print "New files", diff
        old_stat = new_stat
        old_files = new_files

With inotify I think this is all handled for you, and you put your code into process_IN_CREATE() which gets called when a new file is available.

One bit of trickiness - how does the watcher know that the upload is complete? What happens if the upload is canceled part-way through uploading? This could be as simple as having the web server do a rename() to use one extension during upload and another extension when done.

Once you know the file, use subprocess.Popen(conversion_program, "new_filename") or os.system("conversion_program new_filename &") to spawn off the conversion in a new process which does the conversion. You'll need to handle things like error reporting, as when the input isn't in the right format. It should also clean up, meaning that once the conversion is done it should remove the input file from consideration. This might be as easy as deleting the file.

You'll also need to worry about restarting any conversions which were killed. If the machine does down, how does the restarted watcher know which data file conversions were also killed and need to be restarted. But this might be doable as a manual step.