hosting simple python scripts in a container to handle concurrency, configuration, caching, etc.

Question

My first real-world Python project is to write a simple framework (or re-use/adapt an existing one) which can wrap small python scripts (which are used to gather custom data for a monitoring tool) with a "container" to handle boilerplate tasks like:

• fetching a script's configuration from a file (and keeping that info up to date if the file changes and handle decryption of sensitive config data)
• running multiple instances of the same script in different threads instead of spinning up a new process for each one
• expose an API for caching expensive data and storing persistent state from one script invocation to the next

Today, script authors must handle the issues above, which usually means that most script authors don't handle them correctly, causing bugs and performance problems. In addition to avoiding bugs, we want a solution which lowers the bar to create and maintain scripts, especially given that many script authors may not be trained programmers.

Below are examples of the API I've been thinking of, and which I'm looking to get your feedback about.

A scripter would need to build a single method which takes (as input) the configuration that the script needs to do its job, and either returns a python object or calls a method to stream back data in chunks. Optionally, a scripter could supply methods to handle startup and/or shutdown tasks.

HTTP-fetching script example (in pseudocode, omitting the actual data-fetching details to focus on the container's API):

def run (config, context, cache) : 
    results = http_library_call (config.url, config.http_method, config.username, config.password, ...) 
    return { html : results.html, status_code : results.status, headers : results.response_headers }

def init(config, context, cache) : 
     config.max_threads = 20  # up to 20 URLs at one time (per process) 
     config.max_processes = 3  # launch up to 3 concurrent processes 
     config.keepalive = 1200  # keep process alive for 10 mins without another call
     config.process_recycle.requests = 1000  # restart the process every 1000 requests (to avoid leaks) 
     config.kill_timeout = 600  # kill the process if any call lasts longer than 10 minutes 

Database-data fetching script example might look like this (in pseudocode):

def run (config, context, cache) : 
    expensive = context.cache["something_expensive"] 
    for record in db_library_call (expensive, context.checkpoint, config.connection_string) : 
        context.log (record, "logDate")  # log all properties, optionally specify name of timestamp property 
        last_date = record["logDate"] 
    context.checkpoint = last_date  # persistent checkpoint, used next time through 

def init(config, context, cache) : 
    cache["something_expensive"] = get_expensive_thing() 

def shutdown(config, context, cache) : 
    expensive = cache["something_expensive"] 
    expensive.release_me()

Is this API appropriately "pythonic", or are there things I should do to make this more natural to the Python scripter? (I'm more familiar with building C++/C#/Java APIs so I suspect I'm missing useful Python idioms.)

Specific questions:

• is it natural to pass a "config" object into a method and ask the callee to set various configuration options? Or is there another preferred way to do this?
• when a callee needs to stream data back to its caller, is a method like context.log() (see above) appropriate, or should I be using yield instead? (yeild seems natural, but I worry it'd be over the head of most scripters)
• My approach requires scripts to define functions with predefined names (e.g. "run", "init", "shutdown"). Is this a good way to do it? If not, what other mechanism would be more natural?
• I'm passing the same config, context, cache parameters into every method. Would it be better to use a single "context" parameter instead? Would it be better to use global variables instead?
• Finally, are there existing libraries you'd recommend to make this kind of simple "script-running container" easier to write?

Answer 1

Have a look at SQL Alchemy for dealing with database stuff in python. Also to make script writing easier for dealing with concurrency look into Stackless Python.