Python memory leaks

Question

I have a long-running script which, if let to run long enough, will consume all the memory on my system.

Without going into details about the script, I have two questions:

1. Are there any "Best Practices" to follow, which will help prevent leaks from occurring?
2. What techniques are there to debug memory leaks in Python?

Answer 1

Have a look at this article: Tracing python memory leaks

Also, note that the garbage collection module actually can have debug flags set. Look at the set_debug function. Additionally, look at this code by Gnibbler for determining the types of objects that have been created after a call.

Answer 2

Not sure about "Best Practices" for memory leaks in python, but python should clear it's own memory by it's garbage collector. So mainly I would start by checking for circular list of some short, since they won't be picked up by the garbage collector.

Answer 3

You should specially have a look on your global or static data (long living data).

When this data grows without restriction, you can also get troubles in Python.

The garbage collector can only collect data, that is not referenced any more. But your static data can hookup data elements that should be freed.

An other problem can be memory cycles, but at least in theory the Garbage collector should find and eliminate cycles -- at least as long as they are not hooked on some long living data.

What kinds of long living data are specially troublesome? Have a good look on any lists and dictionaries -- they can grow without any limit. In dictionaries you might even don't see the trouble coming since when you access dicts, the number of keys in the dictionary might not be of big visibility to you ...

Answer 4

I have found this recipe helpful.