How can I find memory leaks in my Python program?

Question

I've got a fairly complex (about 20,000) line Python program which after some development has started consuming increasing amounts of memory when it runs. What are the best tools and techniques for finding out what all the memory is being used for?

Usually this comes down to either unexpectedly keeping references to objects, or extension module bugs (which isn't particularly likely since we've been using the Python 2.4 installation for a while).

We are using various third party libraries such as Twisted, Twisted Conch and MySQLdb.

Answer 1

There was already a question on this topic.

Answer 2

Python's memory is managed by a garbage collector. In general, there shouldn't be a problem with memory leaking (definitely not for Python2.5 and above), unless you happen to be writing extension modules in C/C++. In that case, Valgrind (Blog post -http://bruynooghe.blogspot.com/2008/12/finding-memory-leaks-in-python.html) might be helpful. I found that this person - http://mg.pov.lt/blog/hunting-python-memleaks has used PDB and matplotlib to trace a memory leak. I hope this helps, I have no experience fixing Python memory leaks.

Answer 3

Generally, failing to close cursors is one of the most common kinds of memory leaks. The garbage collector can't see the MySQL resources involved in the cursor. MySQL doesn't know that the Python side was released unless the close() method is called explicitly.

Rule of thumb. Open, use and close cursors in as short a span of code as you can manage.