lazy event publish subscribe in python

Question

I need an event messaging system in my google app engine application.

and i was referring to following python library.

http://pubsub.sourceforge.net/apidocs/concepts.html

my question is , is it must that the listener function i want to execute must be imported ( or exist otherwise) somewhere in to the execution path in order to run it on event ?

There are many lots of event, and I want to make it as lazy loaded as possible.

what could be the work around ?

is there any lazy event publish subscribe framework in python ?

Answer 1

tipfy (an App-Engine specific micro framework) has lazy loading, but only for the specific "events" that are web requests your code is serving. Other web frameworks have it too, but tipfy is small and simple enough to easily study and imitate its sources for the purpose.

So, if you can't find a richer event framework that's exactly to your taste because of the "lazy loading" issue, you could pick one which requires registration/subscription of callable objects, and allow strings naming functions to be registered as well, just as tipfy does. The function thus named, of course, would be loaded just in time if an when needed to serve some event.

Let me exemplify with some simplified, hypothetical code. Say that you have an event framework that includes something like:

import collections
servers = collections.defaultdict(list)

def register(eventname, callable):
    servers[eventname].append(callable)

def raise(eventname, *a, **k):
    for s in servers.get(eventname, ()):
        s(*a, **k)

The internals of any real-world event framework will be richer, of course, but something like this will be discernible at the lowest layers of it.

So, this requires the callable to be loaded at registration time... and yet, even without touching the internals of your framework, you can easily extend it. Consider:

import sys

class LazyCall(object):
    def __init__(self, name):
        self.name = name
        self.f = None
    def __call__(self, *a, **k):
        if self.f is None:
            modname, funname = self.name.rsplit('.', 1)
            if modname not in sys.modules:
                __import__(modname)
            self.f = getattr(sys.modules[modname], funname)
        self.f(*a, **k)

You'll want better error handling &c, of course, but this is the gist of it: wrap the string naming the function (e.g. 'package.module.func') into a wrapper object that knows how to lazily load it. Now, register(LazyCall('package.module.func')) will register, in the untouched framework, such a wrapper -- and lazy-load it on request.

This use case, btw, could be used as a reasonably good example of a Python idiom that some obstreperous fools claim, loudly and stridently, doesn't exist, or shouldn't exist, or something: an object dynamically changing its own class. Use cases for this idiom are to "cut the middleman" for objects that exist in one of two states, with the transition from the first to the second being irreversible. Here, the first state of a lazy caller is "I know the function's name but don't have the object", the second one is "I know the function object". Since moving from the first to the second is irreversible, you can cut the small overhead of testing every time (or the indirection overhead of the Strategy design pattern), if you wish:

class _JustCallIt(object):
    def __call__(self, *a, **k):
        self.f(*a, **k)

class LazyCall(object):
    def __init__(self, name):
        self.name = name
        self.f = None
    def __call__(self, *a, **k):
        modname, funname = self.name.rsplit('.', 1)
        if modname not in sys.modules:
            __import__(modname)
        self.f = getattr(sys.modules[modname], funname)
        self.__class__ = _JustCallIt
        self.f(*a, **k)

The gain here is modest, as it basically just cuts one if self.f is None: check from each call; but it's a real gain, without real downsides except for causing the previously named obstreperous fools to flail into their typical angry and mindless frenzy (if you count that as a downside).

Anyway, the implementation choice is up to you, not to me -- or, lucky you, to them;-).

As is one design choice: whether to patch register itself to directly accept string arguments (and wrap them as needed), basically as tipfy does, or go for the explicit wrapping at the registration site, leaving register (or subscribe or however it's called) pristine. I don't set much weight by the "explicit is better than implicit" mantra in this particular case, since something like

register(somevent, 'package.module.function')

is quite as explicit as

register(somevent, LazyCall('package.module.function'))

i.e., it is quite clear what's going on, and it's arguably cleaner / more readable.

Nevertheless, it is really nice that the explicit wrapping approach leaves the underlying framework untouched: wherever you could pass a function, you can now pass the name of that function (as a string naming packages, module, and the function itself), seamlessly. So, were I retrofitting existing frameworks, I'd go for the explicit approach.

Finally, if you want to register callables that are not functions but (for example) instances of certain classes, or bound methods of such instances, you can enrich LazyCall into variants such as LazyInstantiateAndCall &c for the purpose. The architecture becomes a tad more complex, of course (since you want ways to instantiate new objects and ways to identify already existing ones, for example), but by delegating that work to a well designed system of factories, it shouldn't be too bad. However, I'm not getting any deeper in such refinements, since this answer is already rather long, and anyway, in many cases, the simple "name a function" approach should suffice!-)