basic python server using spawn/threads....

Question

Hi.

Got a problem that I'm facing. and it should be pretty simple.

I have an app that places data into a dir "A". The data will be a series of files.

I want to have a continually running server, that does a continual look at the dir, and on seeing a completed file in the dir, the server spawns/forks/creates a thread (not sure of the exact word/tech in python) that then performs some work.

Basically, I'm going to be doing an include/execfile("foo") of an external file in the thread, to perform work, based on the file in the dir "A".

I want to be able to have the multiple threads running at the same time. So i'm looking to run the entire process as fast as possible, and implementing threads/spawn/forked process should allow me to have multiple threads running in parallel. There's no communication between the different work processes.

I've seen various examples using twisted, etc.. but I think I'm over thinking this..

Any simple/complete example that I can play with would be great!! (pointers to samples on the 'net would also be cool...

thanks...

Answer 1

In Python, you should consider using the multiprocessing module instead of threads, especially if you have a multicore machine:

multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows

Also consult the following for examples and an introduction.

• multiprocessing Basics
• Communication between processes with multiprocessing