Python Performance - have you ever had to rewrite in something else?

Question

Has anyone ever had code in Python, that turned out not to perform fast enough?

I mean, you were forced to choose another language because of it?

We are investigating using Python for a couple of larger projects, and my feeling is that in most cases, Python is plenty fast enough for most scenarios (compared to say, Java) because it relies on optimized C routines.

I wanted to see if people had instances where they started out in Python, but ended up having to go with something else because of performance.

Thanks.

Answer 1

Yes, I have. I wrote a row-count program for a binary (length-prefixed rather than delimited) bcp output file once and ended up having to redo it in C because the python one was too slow. This program was quite small (it only took a couple of days to re-write it in C), so I didn't bother to try and build a hybrid application (python glue with central routines written in C) but this would also have been a viable route.

A larger application with performance critical bits can be written in a combination of C and a higher level language. You can write the performance-critical parts in C with an interface to Python for the rest of the system. SWIG, Pyrex or Boost.Python (if you're using C++) all provide good mechanisms to do the plumbing for your Python interface. The C API for python is more complex than that for Tcl or Lua, but isn't infeasible to build by hand. For an example of a hand-built Python/C API, check out cx_Oracle.

This approach has been used on quite a number of successful applications going back as far as the 1970s (that I am aware of). Mozilla was substantially written in Javascript around a core engine written in C. Several CAD packages, Interleaf (a technical document publishing system) and of course EMACS are substantially written in LISP with a central C, assembly language or other core. Quite a few commercial and open-source applications (e.g. Chandler or Sungard Front Arena) use embedded Python interpreters and implement substantial parts of the application in Python.

EDIT: In rsponse to Dutch Masters' comment, keeping someone with C or C++ programming skills on the team for a Python project gives you the option of writing some of the application for speed. The areas where you can expect to get a significant performance gain are where the application does something highly iterative over a large data structure or large volume of data. In the case of the row-counter above it had to inhale a series of files totalling several gigabytes and go through a process where it read a varying length prefix and used that to determine the length of the data field. Most of the fields were short (just a few bytes long). This was somewhat bit-twiddly and very low level and iterative, which made it a natural fit for C.

Many of the python libraries such as numpy, cElementTree or cStringIO make use of an optimised C core with a python API that facilitates working with data in aggregate. For example, numpy has matrix data structures and operations written in C which do all the hard work and a Python API that provides services at the aggregate level.

Answer 2

I used to prototype lots of things in python for doing things like log processing. When they didn't run fast enough, I'd rewrite them in ocaml.

In many cases, the python was fine and I was happy with it. In some cases, as it started approaching 23 hours to do a days' logs, I'd get to rewriting. :)

I would like to point out that even in those cases, I may have been better off just profiling the python code and finding a happier python implementation.

Answer 3

This is a much more difficult question to answer than people are willing to admit.

For example, I may be that I am able to write a program that performs better in Python than it does in C. The fallacious conclusion from that statement is "Python is therefore faster than C". In reality, it may be because I have much more recent experience in Python and its best practices and standard libraries.

In fact no one can really answer your question unless they are certain that they can create an optimal solution in both languages, which is unlikely. In other words "My C solution was faster than my Python solution" is not the same as "C is faster than Python"

I'm willing to bet that Guido Van Rossum could have written Python solutions for adam and Dustin's problems that performed quite well.

My rule of thumb is that unless you are writing the sort of application that requires you to count clock cycles, you can probably achieve acceptable performance in Python.

Answer 4

You can always write parts of your application in Python. Not every component is equally important for performance. Python integrates easily with C++ natively, or with Java via Jython, or with .NET via IronPython.

By the way, IronPython is more efficient than the C implementation of Python on some benchmarks.

Answer 5

I've been working for a while now, developing an application that operate on large structured data, stored in several-gigabytes-thick-database and well, Python is good enough for that. The application has GUI client with a multitude of controls (lists, trees, notebooks, virtual lists and more), and a console server.

We had some performance issues, but those were mostly related more to poor algorithm design or database engine limitations (we use Oracle, MS-SQL, MySQL and had short romance with BerkeleyDB used for speed optimizations) than to Python itself. Once you know how to use standard libraries (written in C) properly you can make your code really quick.

As others say - any computation intensive algorithm, code that depends on bit-stuffing, some memory constrained computation - can be done in raw C/C++ for CPU/memory saving (or any other tricks), but the whole user interaction, logging, database handling, error handling - all that makes the application actually run, can be written in Python and it will maintain responsiveness and decent overall performance.

Answer 6

While at uni we were writing a computer vision system for analysing human behaviour based on video clips. We used python because of the excellent PIL, to speed up development and let us get easy access to the image frames we'd extracted from the video for converting to arrays etc.

For 90% of what we wanted it was fine and since the images were reasonably low resolution the speed wasn't bad. However, a few of the processes required some complex pixel-by-pixel computations as well as convolutions which are notoriously slow. For these particular areas we re-wrote the innermost parts of the loops in C and just updated the old Python functions to call the C functions.

This gave us the best of both worlds. We had the ease of data access that python provides, which enabled to develop fast, and then the straight-line speed of C for the most intensive computations.

Answer 7

Whenever I find a Python bottleneck, I rewrite that code in C as a Python module.

For example, I have some hardware that sends image pixels as 4-byte 0RGB. Converting 8M from 0RGB to RGB in Python takes too long so I rewrote it as a Python module.

Writing Python (or other higher level languages) is much faster than writing in C so I use Python until I can't.

Answer 8

Not so far. I work for a company that has a molecular simulation engine and a bunch of programs written in python for processing the large multi-gigabyte datasets. All of our analysis software is now being written in Python because of the huge advantages in development flexibility and time.

If something is not fast enough we profile it with cProfile and find the bottlenecks. Usually there are one or two functions which take up 80 or 90% of the runtime. We then take those functions and rewrite them in C, something which Python makes dead easy with its C API. In many cases this results in an order of magnitude or more speedup. Problem gone. We then go on our merry way continuing to write everything else in Python. Rinse and repeat...

For entire modules or classes we tend to use Boost.python, it can be a bit of a bear but ultimately works well. If it's just a function or two, we sometimes inline it with scipy.weave if the project is already using scipy.

Answer 9

No, I've never had to rewrite. In fact, I started using Python in Maya 8.5. Before Maya 8, the only scripting language available was the built in MEL (Maya Expression Language). Python is actually faster than the built in language that it wraps.

Python's ability to work with complex data types also made it faster because MEL can only store single dimensional arrays (and no pointers). This would require multi-dimensional arrays be faked by either using multiple parallel arrays, or by using slow string concatenation.

Answer 10

This kind of question is likely to start a religious war among language people so let me answer it a little bit differently.

For most cases in today's computing environments your choice of programming language should be based on what you can program efficiently, program well and what makes you happy not the performance characteristics of the language. Also, optimization should generally be the last concern when programming any system.

The typical python way to do things is to start out writing your program in python and then if you notice the performance suffering profile the application and attempt to optimize the hot-spots in python first. If optimizing the python code still isn't good enough the areas of the code that are weighing you down should be re-written as a python module in C. If even after all of that your program isn't fast enough you can either change languages or look at scaling up in hardware or concurrency.

That's the long answer, to answer your question directly; no, python (sometimes with C extensions) has been fast enough for everything I need it to do. The only time I really dip into C is to get access to stuff that donesn't have python bindings.

Edit: My background is a python programmer at a large .com where we use python for everything from the front-end of our websites all the way down to all the back-office systems. Python is very much an enterprise-grade language.

Answer 11

A month ago i had this little program written in Python (for work) that analyzes logs. When then number of log files grew, the program begun to be very slow and i thought i could rewrite it in Java.

I was very interesting. It took a whole day to migrate the same algorithm from Python to Java. At the end of the day, a few benchmark trials showed me clearly that the Java program was some 20% / 25% slower than its Python counterpart. It was a surprise to me.

Writing for the second time the algorithm also showed me that some optimization was possible. So in two hours i completely rewrote the whole thing in Python and it was some 40% faster than the original Python implementation (hence orders of time faster than the Java version I had).

So:

1. Python is a slow language but still it can be faster, for certain tasks, that other supposedly faster languages

2. If you have to spend time writing something in a language whose execution is faster but whose development time is slower (most languages), consider using the same time to analyze the problem, search for libraries, profile and then write better Python code.

Answer 12

I once had to write a pseudo-random number generator for a simulator. I wrote it in Python first, but Python proved to be way too slow; I ended up rewriting it in C, and even that was slow, but not nearly as slow as Python.

Luckily, it's fairly easy to bridge Python and C, so I was able to write the PRNG as a C module and still write the rest of the simulator in Python.

Answer 13

The following link provides an on going comparison between a number of computer languages. It should give you an idea of some of Python's strengths and weaknesses across different problem domains.

Computer Language Benchmarks Game

Answer 14

I'm in the process of rewriting the Perl program OpenKore in Python under the name Erok (reverse of the original Kore). So far, Python is proving to be an overall better language, especially because of its powerful string parsing functions that don't require the use of regular expressions, which really speeds up a lot of its file parsing.

Answer 15

Adding my $0.02 for the record.

My work involves developing numeric models that run over 100's of gigabytes of data. The hard problems are in coming up with a revenue-generating solution quickly (i.e. time-to-market). To be commercially successful the solution also has to execute quickly (compute the solution in minimal amounts of time).

For us Python has proven to be an excellent choice to develop solutions for the reasons commonly cited: fast development time, language expressiveness, rich libraries, etc. But to meet the execution speed needs we've adopted the 'Hybrid' approach that several responses have already mentioned.

1. Using numpy for computationally intense parts. We get within 1.1x to 2.5x the speed of a 'native' C++ solution with numpy with less code, fewer bugs, and shorter development times.
2. Pickling (Python's object serialization) intermediate results to minimize re-computation. The nature of our system requires multiple steps over the same data, so we 'memorize' the results and re-use them where possible.
3. Profiling and choosing better algorithms. It's been said in other responses, but I'll repeat it: we whip-out cProfile and try to replace hot-spots with a better algorithm. Not applicable in all cases.
4. Going to C++. If the above fails then we call a C++ library. We use PyBindGen to write our Python/C++ wrappers. We found it far superior to SWIG, SIP, and Boost.Python as it produces direct Python C API code without an intermediate layer.

Reading this list you might think "What a lot of re-work! I'll just do it in [C/C++/Java/assembler] the first time around and be done with it."

Let me put it into perspective. Using Python we were able to produce a working revenue-generating application in 5 weeks that, in other languages, had previously required 3 months for projects of similar scope. This includes the time needed to optimize the Python parts we found to be slow.

Answer 16

I am developing in python for several years now. Recently i had to list all files in a directory and build a struct with filename, size, attributes and modification date. I did this with os.listdir and os.stat. The code was quite fast, but the more entries in the directories, the slower my code became comapred to other filemanagers listing the same directory, so i rewrote the code using SWIG/C++ and was really surprised how much faster the code was.

Answer 17

While implementing a rather high performance specialized memcache server for certain datatype, storage backend would be more memory efficient and lookup time could be dramatically decreased by bit wise lookup operations (i.e: O(1) lookups).

I wrote all the protocol implementation and event driven daemon part with Python within 2 days, giving us enough time to test on functionality and focusing on performance while team was validating protocol conformance and other bits.

Given the the tools like Pyrex, implementing C extensions for Python is next to trivial for any developer a bit experienced in C. I rewrote the Radix Tree based storage backend with C and made it a Python module with Pyrex within a day. Memory usage for 475,000 prefixes went down from 90MB to 8MB. We got a 1200% jump in the query performance.

Today the subject application is running with pyevent (python interface for libevent) and the new storage backend handles 8000 queries per second on a modest single core server, running as a single process daemon (thanks to libevent) consuming less than 40MB of memory (including the Python interpreter) while handling 300+ simultaneous connections.

It's now deployed country wide in 65 cities. Answering more than 13 million queries a day.

That's a project designed and implemented to production quality less than 5 days. Without Python and Pyrex, that wouldn't be possible.

We could have troubleshoot the performance problem by just using more powerful servers and switch to a multiprocess/multi-instance model while complicating the code and administration tasks, accompanied with much larger memory footprint. There's a fine trade off in these kind of situations. If you have people on board with enough experience on C and agile development methods, Python house is a heaven for high performance, cost effective applications. If not, you can always opt-in for more computational horsepower.

I strongly believe you're on the right decision track to go with Python.

Answer 18

I generally don't rewrite to C before I :

• profile
• rewrite with bette algorithms (generally this is enough)
• rewrite python code with low level performance in mind (but never to the point of having non pythonic / non readable code)
• spend some time rechecking a library cannot do this (first in stdlib, or an external lib)
• tried psyco / other implementations (rarely achieves to get a REAL speed boost in my case)

Then sometimes I created a shared library to do heavy matrix computation code (which couldn't be done with numarray) and called it with ctypes :

• simple to write/build/test a .so / dll in pure C,
• simple to encapsulate the C to python function (ie. you don't if you use basic datatypes since ctypes does all the work of calling the right arguments for you) and certainly fast enough then .

Answer 19

Yes, twice:

• An audio DSP application I wound up completely rewriting in C++ because I couldn't get appropriate performance in Python; I don't consider the Python implementation wasted because it let me prototype the concept very easily, and the C++ port went smoothly because I had a working reference implementaton.

• A procedural graphic rendering project, where generating large 2D texture maps in Python was taking a long time; I wrote a C++ DLL and used ctypes/windll to use it from Python.