views:

50

answers:

2

how is this solution I am using now:

I have a 1MB .dbf file in the same directory of all my .py modules. In main.py I have

import tools

In tool.py code is :

the_list_that_never_changes = loadDbf(file).variables['CNTYIDFP'].

So the_list_that_never_changes is only loaded once and is always in memory ready to be used...correct?

+2  A: 

Static files are stored apart from application files. If you need to load data.pkl from main.py, then don't mark it as a static file and it will be accessible by main.py like any other application file.

Reference: Application Configuration's Handlers For Static Files.


Alternative: Why not define the information stored in data.pkl as a global variable in your Python source? Then you don't have to go through the trouble of reading a file and deserializing its pickled contents, and it will be a bit faster too. This will also make it easy to take advantage of app caching - your data will be loaded once, and then cached for use by subsequent requests.

David Underhill
Your alternative won't necessarily be any faster: Python still has to load and parse the module you embedded the pickle (or the textual representation in), which needn't be any quicker than loading the pickle instead.
Nick Johnson
I changed my question above now that I realise I can simply add data files within same dirextory as py modules
indiehacker
Correct - `the_list_that_never_changes` will only be evaluated once *per instance* due to app caching. Once loaded, it will stay in memory until the instance is terminated (due to low traffic). When your app spins up a new instance (because of high traffic, or because your app was stopped during a period of no traffic) then this variable will be initialized in the new instance.
David Underhill
OK thanks for clearing that up. I think I got it now.
indiehacker
A: 

Put data.pkl in the same directory with main.py and use something along these lines:

pickle_path = os.path.join(os.path.dirname(__file__), 'data.pkl')
f = open(pickle_path)
data = pickle.load(f)

Do not add data.pkl to app.yaml.

If you read this data often, it might be beneficial to memcache it after unpickling. Then you can read it from memcache, which is usually faster than reading file from disk.

Constantin
I am still curious ..1st: with your solution isn't 'data' now available in memory ? so why would I need it in memcache?
indiehacker
please see new version of question thanks!
indiehacker
@indiehacker, It is not /always/ in memory. Global variables only live as long as instance is running. See http://code.google.com/intl/en/appengine/docs/python/runtime.html#App_Caching
Constantin
sorry for being slow on this .....but i thought even objects in memcache will fall out of it after a while.....so what happens then..where is the data? does it have to be reloaded from the data.pkl file of your example?
indiehacker
Yes, that is true, entries will be evicted from memcache eventually. The pattern is to read from faster cache first. If the entry is not there (first time access or already evicted), then read from slower data source *and* put in cache for future use.From my experience memcached data lives longer than instance-cached data. Probably because memcache service was designed for this kind of usage and instance-cache is more like platform implementation accident.
Constantin