Guidance on optimising Python runtime for embedded systems with low system resources

Question

My team is incorporating the Python 2.4.4 runtime into our project in order to leverage some externally developed functionality.

Our platform has a 450Mhz SH4 application core and limited memory for use by the Python runtime and application.

We have ported Python, but initial testing has highlighted the following hurdles:

a) start-up times for the Python runtime can be as bad as 25 seconds (when importing the libraries concerned, and in turn their dependencies)

b) Python never seems to release memory to the OS during garbage collection - the only recourse is to close the runtime and restart (incurring start-up delays noted above, which often times is impractical)

If we can mitigate these issues our use of Python would be substantially improved. Any guidance from the SO community would be very valuable. Especially from anyone who has knowledge of the intrinsics of how the Python execution engine operates.

Answer 1

Perhaps it is hard to believe, but CPython version 2.4 never releases memory to the OS. This is allegedly fixed in verion Python 2.5.

In addition, performance (processor-wise) was improved in Python 2.5 and Python 2.6 on top of that.

See the C API section in What's new in Python 2.5, look for the item called Evan Jones’s patch to obmalloc

Alex Martelli (whose advice should always be at least considered), says multiprocess is the only way to go to free memory. If you cannot use multiprocessing (module in Python 2.6), os.fork is at least available. Using os.fork in the most primitive manner (fork one work process at the beginning, wait for it to finish, fork a new..) is still better than relauching the interpreter paying 25 seconds for that.