Profile Memory Allocation in Python (with support for Numpy arrays)

Question

I have a program that contains a large number of objects, many of them Numpy arrays. My program is swapping miserably, and I'm trying to reduce the memory usage, because it actually can't finis on my system with the current memory requirements.

I am looking for a nice profiler that would allow me to check the amount of memory consumed by various objects (I'm envisioning a memory counterpart to cProfile) so that I know where to optimize.

I've heard decent things about Heapy, but Heapy unfortunately does not support Numpy arrays, and most of my program involves Numpy arrays.

Answer 1

Can you just save/pickle some of the arrays to disk in tmp files when not using them? That's what I've had to do in the past with large arrays. Of course this will slow the program down, but at least it'll finish. Unless you need them all at once?

Answer 2

Looks like this question has already been answered here --> http://stackoverflow.com/questions/110259/python-memory-profiler

The answers given also includes a graphical browser.

Answer 3

Dealing with large arays in numpy? Consider Numexpr http://www.scipy.org/SciPyPackages/NumExpr

it can reduce your calculation time by a factor of ten.