Is this a memory leak?

Question

I'm using gc module to debug a leak.

It's a gui program and I've hooked this function to a button.

I've set set debug more to gc.SAVE_ALL

> gc.collect()
> 
> print gc.garbage

and this is the output

[(<type '_ctypes.Array'>,), {'__module__': 'ctypes._endian', '__dict__': <attribute '__dict__' of 'c_int_Array_3' objects>, '__weakref__': <attribute '__weakref__' of 'c_int_Array_3' objects>, '_length_': 3, '_type_': <class 'ctypes.c_int'>, '__doc__': None}, <class 'ctypes._endian.c_int_Array_3'>, <attribute '__dict__' of 'c_int_Array_3' objects>, <attribute '__weakref__' of 'c_int_Array_3' objects>, (<class 'ctypes._endian.c_int_Array_3'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), (<type '_ctypes.CFuncPtr'>,), {'__module__': 'ctypes', '__dict__': <attribute '__dict__' of '_FuncPtr' objects>, '__weakref__': <attribute '__weakref__' of '_FuncPtr' objects>, '_flags_': 1, '__doc__': None, '_restype_': <class 'ctypes.c_int'>}, <class 'ctypes._FuncPtr'>, <attribute '__dict__' of '_FuncPtr' objects>, <attribute '__weakref__' of '_FuncPtr' objects>, (<class 'ctypes._FuncPtr'>, <type '_ctypes.CFuncPtr'>, <type '_ctypes._CData'>, <type 'object'>), {}, <cell at 0x10a24b0: Resource object at 0x10e6a50>, <cell at 0x10a2478: dict object at 0x11a4440>, <cell at 0x7f1703949f68: function object at 0x10ec7d0>, <cell at 0x10e2f30: NoneType object at 0x826880>, <cell at 0x10e2d70: NoneType object at 0x826880>, <cell at 0x10e2ef8: str object at 0x7f1703dd5e10>, <cell at 0x10e2de0: dict object at 0x118aaa0>, {'Accept': 'application/json', 'User-Agent': 'couchdb-python 0.6'}, (<cell at 0x10e2f30: NoneType object at 0x826880>, <cell at 0x10a24b0: Resource object at 0x10e6a50>, <cell at 0x10e2de0: dict object at 0x118aaa0>, <cell at 0x10a2478: dict object at 0x11a4440>, <cell at 0x10e2d70: NoneType object at 0x826880>, <cell at 0x10e2ef8: str object at 0x7f1703dd5e10>, <cell at 0x7f1703949f68: function object at 0x10ec7d0>), <function _make_request at 0x10ec7d0>, (1,), {}, <cell at 0x10e2bb0: Resource object at 0x10e6a50>, <cell at 0x10e2e88: dict object at 0x119f360>, <cell at 0x10f0130: function object at 0x10ec578>, <cell at 0x10f01d8: NoneType object at 0x826880>, <cell at 0x10f01a0: NoneType object at 0x826880>, <cell at 0x10f00f8: str object at 0x7f170b05d810>, <cell at 0x10f00c0: dict object at 0x11969a0>, {'Accept': 'application/json', 'User-Agent': 'couchdb-python 0.6'}, (<cell at 0x10f01d8: NoneType object at 0x826880>, <cell at 0x10e2bb0: Resource object at 0x10e6a50>, <cell at 0x10f00c0: dict object at 0x11969a0>, <cell at 0x10e2e88: dict object at 0x119f360>, <cell at 0x10f01a0: NoneType object at 0x826880>, <cell at 0x10f00f8: str object at 0x7f170b05d810>, <cell at 0x10f0130: function object at 0x10ec578>), <function _make_request at 0x10ec578>, (1,), {}, <cell at 0x10f0440: Resource object at 0x10e6a50>, <cell at 0x10f02b8: dict object at 0x11b2d70>, <cell at 0x10f0360: function object at 0x10ec6e0>, <cell at 0x10f0280: NoneType object at 0x826880>, <cell at 0x10f02f0: str object at 0x10ca228>, <cell at 0x10f0408: str object at 0x7f170b05d810>, <cell at 0x10f0050: dict object at 0x11b6370>, {'Accept': 'application/json', 'User-Agent': 'couchdb-python 0.6'}, (<cell at 0x10f0280: NoneType object at 0x826880>]

The gc.garbage list has a lot of items. Does this mean the objects in gc.garbage are leaking or have been collected or will be collected?

Answer 1

From the docs:

gc.garbage

A list of objects which the collector found to be unreachable but could not be freed (uncollectable objects).

So it looks like some kind of leak to me. Now the docs go on to explain the conditions under which this could occur:

Objects that have del() methods and are part of a reference cycle cause the entire reference cycle to be uncollectable, including objects not necessarily in the cycle but reachable only from it. Python doesn’t collect such cycles automatically because, in general, it isn’t possible for Python to guess a safe order in which to run the del() methods. If you know a safe order, you can force the issue by examining the garbage list, and explicitly breaking cycles due to your objects within the list. Note that these objects are kept alive even so by virtue of being in the garbage list, so they should be removed from garbage too. For example, after breaking cycles, do del gc.garbage[:] to empty the list. It’s generally better to avoid the issue by not creating cycles containing objects with del() methods, and garbage can be examined in that case to verify that no such cycles are being created.

Now having the DEBUG_SAVEALL flag set makes all of your garbage leak. From the same source:

gc.DEBUG_SAVEALL

When set, all unreachable objects found will be appended to garbage rather than being freed. This can be useful for debugging a leaking program.

So, again, yes, that list is leaked memory. But you told it to leak all that!

Answer 2

In other languages, I've used a heap profiler with great success to track down memory leaks. I've never used one in Python, but Heapy seems like it'd be worth trying.

Under their "Data Processing" section, try this feature out:

• Calculation of the 'dominated' set from a set of root objects which yields the set of objects that would be deallocated if the root objects were deallocated.

If it is like other tools, you should be able to drill down. Follow the objects with the biggest 'dominated set' until you find something that seems too big, that's likely a leak.