"Pythonic" multithreaded (Concurrent) language

Question

Hi,

I now primarily write in python, however I am looking for a language that is more thread friendly (not JAVA,C#,C or C++).

Python's threads are good when they are IO bound but it's coming up short when I am doing something CPU intensive.

Any ideas?

Thanks, James

Answer 1

to overcome GIL, you can try interpreting python language with jython instead of cpython

Answer 2

Python falls short when it comes to CPU intensive tasks because Python isn't a very efficient language. Many "dynamic" languages have that problem, since the compiler is limited in what it can assume about the program at compile-time. Method-calls, for example, may have to be looked up every single time in case someone has replaced the method on the object between each invokation.

I would recommend that you take a look at Erlang, even though it probably isn't "Pythonic" in your sense of the word. :-)

Answer 3

Clojure is pretty fun, if you're into that sort of thing. It's a lisp that runs on the JVM. Apparently it's as fast as Java for a lot of things, despite being dynamically typed *. Java interop is about as convenient as I could imagine possible, though the native clojure libraries are already decent enough that you don't need to hit up Java for most things.

It also adds some "scripting language" sensibilities, like maps (-> Python dicts) and vectors (-> Python lists) to help reduce the likelihood of parenthetical paroxysm.

Oh right, concurrency. It uses a software transactional memory system which is pretty interesting in and of itself. Extra, extra: Read all about it.

*: I think you may need to use "type hinting" to get Java-like speed for a lot of tasks. This is made quite convenient in Clojure.

Answer 4

Stackless Python as used by the EVE online developers might fit.

Stackless Python is an enhanced version of the Python programming language. It allows programmers to reap the benefits of thread-based programming without the performance and complexity problems associated with conventional threads. The microthreads that Stackless adds to Python are a cheap and lightweight convenience, which if used properly, can not only serve as a way to structure an application or framework, but by doing so improve program structure and facilitate more readable code.

Answer 5

boo's syntax is Python-inspired. The programming language is a bit different, though: It's strongly typed with type-inference, and it's most imporant feature is probably the open compiler pipeline, i.e. the ability to create syntactic macros (in the LISP sense of the word "macro", not in the C sense of "preprocessor macro").

And, obviously, IronPyton is quite pythonic and about as good at threading as other .NET languages.

Answer 6

Before anything else, what programming problem are you trying to solve? Has the GIL actually become a bottleneck yet? (If you're not sure, it probably hasn't.)

Without knowing this, you risk looking for a nail that fits your hammer, instead of the other way around.

If you really, really know that the GIL has become a bottleneck, and that you need to crunch numbers across multiple processors, then you want to consider:

1. If you have known CPU-critical sections, compile them with C type declarations using Pyrex/Cython, if you can. This will firstly make them much more CPU-efficient than Python bytecode interpretation, and as a bonus, allow you to release the GIL around sections that don't need it. (Moral: turn your donkeys into racehorses before trying to parallelize them.)
2. It should go without saying, but if you're using any CPU-intensive extension modules, make sure they're not already releasing the GIL for you.
3. For almost everything else, use the built-in multiprocessing module. This gives you roughly the same API and advantages as threading, but provides you with true process-level concurrency: among other things, this allows you to easily run your CPU-intensive code on multiple machines in concert, which is critical if your problem actually gets big.

Answer 7

Take the hint. CPU intensive programs can also be made into multiple processes. Multiple processes and a pipeline to pass status around can often have outstanding performance.

Rather than fish around randomly for other languages, do this.

1. Decompose the problem into a pipeline of steps that can be done concurrently.

   In the shell, the top-level script is this: a.py | b.py | c.py | d.py...

2. Write each step as a very small Python loop that reads from sys.stdin and writes to sys.stdout. Interesting, this is the default for raw_input() and print() making things simple.

3. Measure the performance.

You'll -- correctly -- spend all your time designing your algorithm. You'll spend little time coding or learning a new language. You'll trivially tie up every core on every CPU available to you. You'll spend no time on thread synchronization or other foolishness.

This kind of thing works very, very well for "CPU Intensive" applications.

Answer 8

You could just use python multiprocessing it mirrors the API of threading but runs separate processes. Might not be implemented for non-posix. For general CPU intensive problems you could always try Stackless Python (mentioned already) or Pyrex.