How can I measure the overall memory requirements of a Python program

Question

I have a financial pricing application written in Python 2.4.4 which runs as an Excel plugin. Excel has a 1GB memory limit for all addins, so if any addin process tries to allocate more than 1Gb in total it will cause Excel to crash.

I've recently made a change to the program which may have changed the overall memory requirements of the program. I'd like to work out if anything has changed sigifnicantly, and if not I can reassure my management that there's no chance of a failure due to an increase in memory usage.

Incidentally, it's possible to run the same function which runs in Excel from a command-line:

Effectively I can provide all the arguments that would be generated from Excel from the regular windows prompt. This means that if I have a method to discover the memory requirements of the process on it's own I can safely infer that it will use a similar amount when run from Excel.

I'm not interested in the detail that a memory profiler might give: I do not need to know how much memory each function tries to allocate. What I do need to know is the smallest amount of memory that the program will require in order to run and I need to guarantee that if the program is run within a limit of 1Gb it will run OK.

Any suggestions as to how I can do this?

Platform is Windows XP 32bit, python 2.4.4

Answer 1

Simplest solution might just be to run your app from the command line and use Task Manager to see how much memory it's using.

Edit: For something more complicated have a look at this question / answer http://stackoverflow.com/questions/282194/how-to-get-memory-usage-under-windows-in-c

Edit: Another option. Get Process Explorer. It's basically Task Manager on steroids. It'll record peak usage of a process for you. (it's free so you don't have to worry about cost)

Answer 2

I don't believe XP keeps track of the peak memory requirements of a process (the way Linux does in /proc/pid/status for example). You can use third-party utilities such as this one and set it to "poll" the process very frequently to get a good chance of grabbing the correct peak.

A better approach, though it doesn't answer your title question, is to try running the process under a job object with a SetInformationJobObject with a JOBOBJECT_EXTENDED_LIMIT_INFORMATION structure having a suitable ProcessMemoryLimit (e.g 1GB max virtual memory) and the right flag in JOBOBJECT_BASIC_LIMIT_INFORMATION to activate that limit. This way, if the process runs correctly, you will KNOW it has never used more than 1 GB of virtual memory -- otherwise, it will crash with an out-of-memory error.

This programmatic approach also allows measurement of peak values, but it's complicated enough, compared with using already packaged tools, that I don't think it can be recommended for that purpose. I do not know of packaged tools that allow you to use job objects to specifically limit the resources available to processes (though I wouldn't be surprised to hear that some are available, be that freely or commercially, I've never come upon one myself).