Issues with time.sleep and Multithreading in Python

Question

I am having an issue with the time.sleep() function in python. I am running a script that needs to wait for another program to generate txt files. Although, this is a terribly old machine, so when I sleep the python script, I run into issues with the other program not generating files. Is there any alternatives to using time.sleep()? I thought locking the thread might work but essentially it would just be a loop of locking the thread for a couple of seconds. I'll give some pseudo code here of what I'm doing.

While running:
    if filesFound != []:
         moveFiles
    else:
       time.sleep(1)

Answer 1

Check the documentation for the "subprocess" module on the standard lib. The Popen class in there allows various ways of dealing with any sub-process you call. Probably you won't even need to use threads when you find out how to properly use subprocess.

The docuemntation is available in the module itself, just do

import subprocess
help(subprocess)

Answer 2

One way to do a non-blocking wait is to use threading.Event:

import threading
dummy_event = threading.Event()
dummy_event.wait(timeout=1)

This can be set() from another thread to indicate that something has completed. But if you are doing stuff in another thread, you could avoid the timeout and event altogether and just join the other thread:

import threading

def create_the_file(completion_event):
    # Do stuff to create the file

def Main():
    worker = threading.Thread(target=create_the_file)
    worker.start()

    # We will stop here until the "create_the_file" function finishes
    worker.join()

    # Do stuff with the file

If you want an example of using events for more fine-grained control, I can show you that...

The threading approach won't work if your platform doesn't provide the threading module. For example, if you try to substitute the dummy_threading module, dummy_event.wait() returns immediately. Not sure about the join() approach.

If you are waiting for other processes to finish, you would be better off managing them from your own script using the subprocess module (and then, for example, using the wait method to be sure the process is done before you do further work).

If you can't manage the subprocess from your script, but you know the PID, you can use the os.waitpid() function. Beware of the OSError if the process has already finished by the time you use this function...

If you want a cross-platform way to watch a directory to be notified of new files, I'd suggest using a GIO FileMonitor from PyGTK/PyGObject. You can get a monitor on a directory using the monitor_directory method of a GIO.File.

Quick sample code for a directory watch:

import gio

def directory_changed(monitor, file1, file2, evt_type):
    print "Changed:", file1, file2, evt_type

gfile = gio.File(".")
monitor = gfile.monitor_directory(gio.FILE_MONITOR_NONE, None)
monitor.connect("changed", directory_changed) 

import glib
ml = glib.MainLoop()
ml.run()