multiprocessing

MPI for multicore ?

With the recent buzz on multicore programming is anyone exploring the possibilities of using MPI ? ...

Python 2.6 multiprocessing.Queue compatible with threads?

I am experimenting with the new multiprocessing module in Python 2.6. I am creating several processes each with its own multiprocessor.JoinableQueue instance. Each process spawns one or more worker threads (subclasses of threading.Thread) which share the JoinableQueue instance (passed in through each Thread's __init__ method). It seem...

Proper way of cancelling accept and closing a Python processing/multiprocessing Listener connection.

Hi! (I'm using the pyprocessing module in this example, but replacing processing with multiprocessing should probably work if you run python 2.6 or use the multiprocessing backport) I currently have a program that listens to a unix socket (using a processing.connection.Listener), accept connections and spawns a thread handling the requ...

How can I handle multiple sockets within a Perl daemon with large memory usage?

I have created a client-server program with Perl using IO::Socket::INET. I access server through CGI based site. My server program will run as daemon and will accept multiple simultaneous connections. My server process consumes about 100MB of memory space (9 large arrays, many arrays...). I want these hashes to reside in memory and shar...

Why don't TKinter windows appear when using multiprocessing on Linux?

I want to spawn another process to display an error message asynchronously while the rest of the application continues. I'm using the multiprocessing module in Python 2.6 to create the process and I'm trying to display the window with TKinter. This code worked okay on Windows, but running it on Linux the TKinter window does not appear if...

Efficient layout for a distributed python server?

If I wanted to have Python distributed across multiple processors on multiple computers, what would my best approach be? If I have 3 eight-core servers, that would mean I would have to run 24 python processes. I would be using the multiprocessing library, and to share objects it looks like the best idea would be to use a manager. I want ...

How should I log while using multiprocessing in Python?

Right now I have a central module in a framework that spawns multiple processes using the Python 2.6 multiprocessing module. Because it uses multiprocessing, there is module-level multiprocessing-aware log, LOG = multiprocessing.get_logger(). Per the docs, this logger has process-shared locks so that you don't garble things up in sys.std...

Python multiprocessing: sharing a large read-only object between processes?

Do child processes spawned via multiprocessing share objects created earlier in the program? I have the following setup: do_some_processing(filename): for line in file(filename): if line.split(',')[0] in big_lookup_object: # something here if __name__ == '__main__': big_lookup_object = marshal.load('file.bi...

multithreading or multiprocessing

I am designing a dedicated syslog-processing daemon for Linux that needs to be robust and scalable and I'm debating multithread vs. multiprocess. The obvious objection with multithreading is complexity and nasty bugs. Multi-processes may impact performance because of IPC communications and context switching. "The Art of Unix Programmi...

Working with multiple processes in Ruby

Is there a module for Ruby that makes it easy to share objects between multiple processes? I'm looking for something similar to Python's multiprocessing, which supports process-safe queues and pipes that can be shared between processes. ...

Multiprocessing or Multithreading?

I'm making a program for running simulations in Python, with a wxPython interface. In the program, you can create a simulation, and the program renders (=calculates) it for you. Rendering can be very time-consuming sometimes. When the user starts a simulation, and defines an initial state, I want the program to render the simulation con...

Followup: Multiprocessing or Multithreading for Python simulation software

Hello folks, this is a follow up to this. (You don't have to read all the answers, just the question) People explained to me the difference between processes and threads. On the one hand, I wanted processes so I could fully exploit all core of the CPU, on the other hand, passing information between processes was less than ideal, and I d...

Python: Locks from `threading` and `multiprocessing` interchangable?

Are the locks from the threading module interchangeable with those from the multiprocessing module? ...

Dynamic processes in Python

I have a question concerning Python multiprocessing. I am trying to take a dataset, break into chunks, and pass those chunks to concurrently running processes. I need to transform large tables of data using simple calculations (eg. electrical resistance -> temperature for a thermistor). The code listed below almost works as desired, but...

Python multiprocessing: Pool of custom Processes

I am subclassing the Process class, into a class I call EdgeRenderer. I want to use multiprocessing.Pool, except instead of regular Processes, I want them to be instances of my EdgeRenderer. Possible? How? ...

Python: Good place to learn about `multiprocessing.Manager`?

I want to learn to use multiprocessing.Manager. I looked at the documentation but it's not easy enough for me. Anyone knows of a good tutorial or something like that? ...

Python Multiprocessing: Sending data to a process

I have subclassed Process like so: class EdgeRenderer(Process): def __init__(self,starter,*args,**kwargs): Process.__init__(self,*args,**kwargs) self.starter=starter Then I define a run method which uses self.starter. That starter object is of a class State that I define. Is it okay that I do this? What happens t...

Using Python multiprocessing while importing a module via file path

I'm writing a program which imports a module using a file path, with the function imp.load_source(module_name,module_path). It seems to cause a problem when I try to pass objects from this module into a Process. An example: import multiprocessing import imp class MyProcess(multiprocessing.Process): def __init__(self,thing): ...

Python multiprocessing on Python 2.6 Win32 (xp)

I tried to copy this example from this Multiprocessing lecture by jesse noller (as recommended in another SO post)[http://pycon.blip.tv/file/1947354?filename=Pycon-IntroductionToMultiprocessingInPython630.mp4] But for some reason I'm getting an error, as though it's ignoring my function definitions: I'm on Windows XP (win32) which I kno...

Python: Plugging wx.py.shell.Shell into a separate process

I would like to create a shell which will control a separate process that I created with the multiprocessing module. Possible? How? EDIT: I have already achieved a way to send commands to the secondary process: I created a code.InteractiveConsole in that process, and attached it to an input queue and an output queue, so I can command t...