views:

60

answers:

1

I've recently uploaded an app that uses django appengine patch and currently have a cron job that runs every two minutes. On each invocation of the worker url it consumes quite a bit of resources

/worker_url  200 7633ms 34275cpu_ms 28116api_ms

That is because on each invocation it does a cold zipimport of all the libraries django etc.

  • How long do the imported modules stay in memory?
  • Is there a way to keep these modules in memory so even if subsequent calls are not within the timeframe that these modules stay in memory they still don't invoke the overhead?
+1  A: 

app engine keeps everything in memory according to normal Python semantics as long as it's serving one or more requests in the same process in the same node; if and when it needs those resources, the process goes away (so nothing stays in memory that it used to have), and new processes may be started (on the same node or different nodes) any time to serve requests (whether other processes serving other requests are still running, or not). This is much like the fast-CGI model: you're assured of normal semantics within a single request but apart from that anything between 0 and N (no upper limit) different nodes may be running your code, each serving sequentially anything between 0 and K (no upper limit) different requests.

There is nothing you can do to "stay in memory" (for zipimported modules or anything else).

For completeness let me mention memcache, which is an explicit hint/request to the app engine runtime to keep something in a special form of memory, a distributed hash table that's shared among all processes running your code -- that's hard though not impossible to use for imported modules (you'd need pretty sophisticated import hooks) and I recommend against the effort needed to develop such hooks because even in presence of such explicit hints the app engine runtime can still at any time choose to eject anything you've stashed away in the cache, anyway.

Rather -- I'm not sure why a cron job in particular would need all of django nor of why you're zipimporting it rather than just using the 1.0.2 that now comes with app engine, per the docs -- care to elaborate? This might be a useful issue for you to optimize.

Alex Martelli