I am writing some multiprocessing code (Python 2.6.4, WinXP) that spawns processes to run background tasks. In playing around with some trivial examples, I am running into an issue where my code just continuously spawns new processes, even though I only tell it to spawn a fixed number. The program itself runs fine, but if I look in Win...
We're considering re-factoring a large application with a complex GUI which is isolated in a decoupled fashion from the back-end, to use the new (Python 2.6) multiprocessing module. The GUI/backend interface uses Queues with Message objects exchanged in both directions. One thing I've just concluded (tentatively, but feel free to confi...
I get the following error when using multiprocessing: Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.6/threading.py", line 525, in __bootstrap_inner self.run() File "/usr/lib/python2.6/threading.py", line 477, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python...
I'm developing a Python project for dealing with computer simulations, and I'm also developing a GUI for it. (The core logic itself does not require a GUI.) The GUI toolkit I use for is wxPython, but I think my question is general enough not to depend on it. The way that the GUI currently works is that it starts the core logic package (...
If a software project supports a version of Python that multiprocessing has been backported to, is there any reason to use threading.Lock over multiprocessing.Lock? Would a multiprocessing lock not be thread safe as well? For that matter, is there a reason to use any synchronization primitives from threading that are also in multiproce...
I would like to know why we prefer to make web servers multi-threaded instead of make it multi-process web servers .... Is it because of legacy issues..... I would like to hear practical reasons as well as theoretical reasons ...
I'm trying to create a program that starts a process pool of, say, 5 processes, performs some operation, and then quits, but leaves the 5 processes open. Later the user can run the program again, and instead of it starting new processes it uses the existing 5. Basically it's a producer-consumer model where: The number of producers va...
I have a python script to download source code from a list of repositories, some of them are big. Sometimes, svn hangs in the middle of a check out. Is there a way to watch over svn process, and so I know it is hang or not? ...
Hi, I'm trying to get python-gasp working on Windows, but when I do import gasp; gasp.begin_graphics() I get the following traceback: File "C:\Python26\lib\site-packages\gasp\backend.py", line 142, in create_screen screen.updater.start() File "C:\Python26\lib\multiprocessing\process.py", line 104, in start self._popen = ...
I need to read some very huge text files (100+ Mb), process every lines with regex and store the data into a structure. My structure inherits from defaultdict, it has a read(self) method that read self.file_name file. Look at this very simple (but not real) example, I'm not using regex, but I'm splitting lines: import multiprocessing ...
I am writing a GUI program using PyQt4. There is a button in my main window and by clicking this button. I hope to launch a background process which is an instance of a class derived from processing.Process. class BackgroundTask(processing.Process): def __init__(self, input): processing.Process.__init__(self) ... ...
I'm having troubles with the multiprocessing module. I'm using a Pool of workers with its map method to load data from lots of files and for each of them i analyze data with with a custom fuction. Each time a file has been processed I would like to have a counter updated so that i can keep track of how many files remains to be processed....
Does anyone know of an easy way to set the niceness value of a Process or Pool when it is created in multiprocessing? ...
i am getting this error when doing database calls in a sub process using multiprocessing library. http://pastie.org/811424 InternalError: current transaction is aborted, commands ignored until end of transaction block this is to a postgres database, using psycopg2 driver in web.py. how ever if i use threading.Thread instead of multip...
I have a pool of processes that need to be executed. I would like to fully utilize the machine, so that all CPUs are executing processes. I do not want to over-subscribe the system, so what i really want is #executing_processes=#cpus at any given moment. I also need to store the stdout,stderr and return code of each completed processes....
While developing a Django app deployed on Apache mod_wsgi I found that in case of multithreading (Python threads; mod_wsgi processes=1 threads=8) Python won't use all available processors. With the multiprocessing approach (mod_wsgi processes=8 threads=1) all is fine and I can load my machine at full. So the question: is this Python beh...
I am new to Python and trying a multiprocessing.pool program to process files, it works fine as long as there are no exceptions. If any of the thread/process gets an exception the whole program waits for the thread snippet of the code: cp = ConfigParser.ConfigParser() cp.read(gdbini) for table in cp.sections(): jobs.append(table) ...
Is putting an object in a multi-processing queue independent from getting an object from it? In other words, will putting an object block the process P1 if another process P2 is getting from it? Update: I am assuming an infinite queue. ...
What would be the advantage(s) (if any) of using 2 Queues over a Pipe to communicate between processes? I am planning on using the multiprocessing python module. ...
Aside from the ease of use of the multiprocessing module when it comes to hooking up processes with communication resources, are there any other differences between spawning multiple processes using multiprocessing compared to using subprocess to launch separate Python VMs ? ...