Does python support multiprocessor/multicore programming?

Question

What is the difference between multiprocessor programming and multicore programming? preferably show examples in python how to write a small program for multiprogramming & multicore programming

Answer 1

You can read about multithreading in python, and threading in general

Multithreading in Python: http://www.devshed.com/c/a/Python/Basic-Threading-in-Python/

Answer 2

If I understand things correctly, Python has something called the GIL (Global Interpreter Lock) that effectively makes it impossible to take advantage of multicores when doing multiple threads in Python.

See eg Guido van Rossum's blog entry on the topic. As far as I know, among the "mainstream" languages only C/C++ and Java have effective support for multicores.

Answer 3

The main difference is how you organize and distribute data. Multicore typically has higher bandwidths between the different cores in a cpu, and multiprocessor needs to involve the bus between the cpus more.

Python 2.6 has gotten multiprocess (process, as in program running) and more synchronization and communication objects for multithreaded programming.

Answer 4

As mentioned in another post Python 2.6 has the multiprocessing module, which can take advantage of multiple cores/processors (it gets around GIL by starting multiple processes transparently). It offers some primitives similar to the threading module. You'll find some (simple) examples of usage in the documentation pages.

Answer 5

There is no such thing as "multiprocessor" or "multicore" programming. The distinction between "multiprocessor" and "multicore" computers is probably not relevant to you as an application programmer; it has to do with subtleties of how the cores share access to memory.

In order to take advantage of a multicore (or multiprocessor) computer, you need a program written in such a way that it can be run in parallel, and a runtime that will allow the program to actually be executed in parallel on multiple cores (and operating system, although any operating system you can run on your PC will do this). This is really parallel programming, although there are different approaches to parallel programming. The ones that are relevant to Python are multiprocessing and multithreading.

In languages like C, C++, Java, and C#, you can write parallel programs by executing multiple threads. The global interpreter lock in the CPython and PyPy runtimes preclude this option; but only for those runtimes. (In my personal opinion, multithreading is dangerous and tricky and it is generally a good thing that Python encourages you not to consider it as a way to get a performance advantage.)

If you want to write a parallel program which can run on multiple cores in Python, you have a few different options:

• Write a multithreaded program using the threading module and run it in the IronPython or Jython runtime.
• Use the processing module, (now included in Python 2.6), to run your code in multiple processes at once.
• Use the subprocess module to run multiple python interpreters and communicate between them.
• Use Twisted and Ampoule. This has the advantage of not just running your code across different processes, but (if you don't share access to things like files) potentially across different computers as well.

No matter which of these options you choose, you will need to understand how to split the work that your program is doing up into chunks that make sense to separate. Since I'm not sure what kind of programs you are thinking of writing, it would be difficult to provide a useful example.

Answer 6

You can actually write programs which will use multiple processors. You cannot do it with threads because of the GIL lock, but you can do it with different process. Either:

• use the subprocess module, and divide your code to execute a process per processor
• have a look at parallelpython module
• if you use python > 2.6 have a look at the multiprocess module.

Answer 7

Always remember, however, that if you also care about performance, using Python is a problem. It's really slow compared for instance to either Java or C#, because it's still interpreted and not JIT-compiled, and the interpreter is not very efficient. To make it fast, most popular recommendations (ranging from manual inlining to writing C extensions) make your program less readable. So using Java or C# might be a much better choice, especially if you need to start from scratch anyway.

Things like Psyco do not, but Psyco has some bugs and does not speed up all code. And still they don't support multithreading.

Jython, instead, is even slower than CPython, while IronPython is not faster than CPython, even if it runs on top of an efficient VM, with JIT-compilation for instance.

Answer 8

If you don't have Python 2.6 (which you don't if you're using Ubuntu Edgy or Intrepid for example), you can use the Google code backported version of multiprocessing. It is part of PyPI, which means you can easily install it using EasyInstall (which is part of the python-setuptools package in Ubuntu).