multiprocessing

Multiprocessing launching too many instances of Python VM

I am writing some multiprocessing code (Python 2.6.4, WinXP) that spawns processes to run background tasks. In playing around with some trivial examples, I am running into an issue where my code just continuously spawns new processes, even though I only tell it to spawn a fixed number. The program itself runs fine, but if I look in Win...

What problems will one see in using Python multiprocessing naively?

We're considering re-factoring a large application with a complex GUI which is isolated in a decoupled fashion from the back-end, to use the new (Python 2.6) multiprocessing module. The GUI/backend interface uses Queues with Message objects exchanged in both directions. One thing I've just concluded (tentatively, but feel free to confi...

Weird unpickling error when using multiprocessing

I get the following error when using multiprocessing: Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.6/threading.py", line 525, in __bootstrap_inner self.run() File "/usr/lib/python2.6/threading.py", line 477, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python...

Python: Separating the GUI process from the core logic process

I'm developing a Python project for dealing with computer simulations, and I'm also developing a GUI for it. (The core logic itself does not require a GUI.) The GUI toolkit I use for is wxPython, but I think my question is general enough not to depend on it. The way that the GUI currently works is that it starts the core logic package (...

Is there any reason to use threading.Lock over multiprocessing.Lock?

If a software project supports a version of Python that multiprocessing has been backported to, is there any reason to use threading.Lock over multiprocessing.Lock? Would a multiprocessing lock not be thread safe as well? For that matter, is there a reason to use any synchronization primitives from threading that are also in multiproce...

Multi-process webserver vs multi threaded web server?

I would like to know why we prefer to make web servers multi-threaded instead of make it multi-process web servers .... Is it because of legacy issues..... I would like to hear practical reasons as well as theoretical reasons ...

Detect if a process is already running and collaborate with it.

I'm trying to create a program that starts a process pool of, say, 5 processes, performs some operation, and then quits, but leaves the 5 processes open. Later the user can run the program again, and instead of it starting new processes it uses the existing 5. Basically it's a producer-consumer model where: The number of producers va...

Watch over another process (svn)

I have a python script to download source code from a list of repositories, some of them are big. Sometimes, svn hangs in the middle of a check out. Is there a way to watch over svn process, and so I know it is hang or not? ...

multiprocessing.Process subclass works on Linux but not Windows

Hi, I'm trying to get python-gasp working on Windows, but when I do import gasp; gasp.begin_graphics() I get the following traceback: File "C:\Python26\lib\site-packages\gasp\backend.py", line 142, in create_screen screen.updater.start() File "C:\Python26\lib\multiprocessing\process.py", line 104, in start self._popen = ...

read multiple files using multiprocessing

I need to read some very huge text files (100+ Mb), process every lines with regex and store the data into a structure. My structure inherits from defaultdict, it has a read(self) method that read self.file_name file. Look at this very simple (but not real) example, I'm not using regex, but I'm splitting lines: import multiprocessing ...

multiprocessing problem [pyqt, py2exe]

I am writing a GUI program using PyQt4. There is a button in my main window and by clicking this button. I hope to launch a background process which is an instance of a class derived from processing.Process. class BackgroundTask(processing.Process): def __init__(self, input): processing.Process.__init__(self) ... ...

Python multiprocessing.

I'm having troubles with the multiprocessing module. I'm using a Pool of workers with its map method to load data from lots of files and for each of them i analyze data with with a custom fuction. Each time a file has been processed I would like to have a counter updated so that i can keep track of how many files remains to be processed....

Multiprocessing and niceness value

Does anyone know of an easy way to set the niceness value of a Process or Pool when it is created in multiprocessing? ...

InternalError: current transaction is aborted, commands ignored until end of transaction block

i am getting this error when doing database calls in a sub process using multiprocessing library. http://pastie.org/811424 InternalError: current transaction is aborted, commands ignored until end of transaction block this is to a postgres database, using psycopg2 driver in web.py. how ever if i use threading.Thread instead of multip...

Simple example of gang scheduling of processes in Python?

I have a pool of processes that need to be executed. I would like to fully utilize the machine, so that all CPUs are executing processes. I do not want to over-subscribe the system, so what i really want is #executing_processes=#cpus at any given moment. I also need to store the stdout,stderr and return code of each completed processes....

Will Python use all processors in thread mode?

While developing a Django app deployed on Apache mod_wsgi I found that in case of multithreading (Python threads; mod_wsgi processes=1 threads=8) Python won't use all available processors. With the multiprocessing approach (mod_wsgi processes=8 threads=1) all is fine and I can load my machine at full. So the question: is this Python beh...

multiprocessing Pool hangs when there is a exception in any of the thread

I am new to Python and trying a multiprocessing.pool program to process files, it works fine as long as there are no exceptions. If any of the thread/process gets an exception the whole program waits for the thread snippet of the code: cp = ConfigParser.ConfigParser() cp.read(gdbini) for table in cp.sections(): jobs.append(table) ...

python multi-processing queue: is putting independent from getting?

Is putting an object in a multi-processing queue independent from getting an object from it? In other words, will putting an object block the process P1 if another process P2 is getting from it? Update: I am assuming an infinite queue. ...

What's the advantage of queues over pipes when communicating between processes?

What would be the advantage(s) (if any) of using 2 Queues over a Pipe to communicate between processes? I am planning on using the multiprocessing python module. ...

Python multiprocessing process vs. standalone Python VM

Aside from the ease of use of the multiprocessing module when it comes to hooking up processes with communication resources, are there any other differences between spawning multiple processes using multiprocessing compared to using subprocess to launch separate Python VMs ? ...