Code Changes While Keeping Large Objects In Memory in Python

Question

I have an application that starts by loading a large pickled trie (173M) from disk and then uses it to do some processing. I'm making frequent changes to the processing part, which is inconvenient because loading the trie takes 15 minutes or so. I'm looking for a way to eliminate the repeated loading during testing, since the trie never changes.

One thing I can't do is use a smaller version of the trie.

Ideas I've had so far are memcached and turning the trie into a web service that accepts a query and returns the data I need.

What I'm looking for is the least-effort path to a situation in which I can repeatedly change and reload the processing code while maintaining access to the in-memory trie. A direct reference to the tree would be preferable since this would require minimal code changes, but really I'm looking to minimize overall effort.

Answer 1

You could try using Pythons built-in reload method or the livecoding project.

Answer 2

This is similar to my recent SO post.

I used the reload method, which worked perfectly for my situation.

import xmlrpc_proxy

# -----------------------------------------------------------------------------------

class XMLRPCContrller(xmlrpc.XMLRPC):

        def __init__(self):
                xmlrpc.XMLRPC.__init__(self)
                self.proxy = xmlrpc_proxy.RequestProxy(self)

       -----------------------------------------------------------------------------------

        def xmlrpc_reloadProxy(self):
                reload(xmlrpc_proxy)
                self.proxy = xmlrpc_proxy.RequestProxy(self)
                return "Reloaded Proxy"

Answer 3

The usual problem with reload is that instances stay bound to the old version of the class. If you are not keeping old instances around, reload is simple and works very well.