Easy Python ASync. Precompiler?

Question

Hi, imagine you have an io heavy function like this:

def getMd5Sum(path):
    with open(path) as f:
        return md5(f.read()).hexdigest()

Do you think Python is flexible enough to allow code like this (notice the $):

def someGuiCallback(filebutton):
    ...
    path = filebutton.getPath()
    md5sum = $getMd5Sum()
    showNotification("Md5Sum of file: %s" % md5sum)
    ...

To be executed something like this:

def someGuiCallback_1(filebutton):
    ...
    path = filebutton.getPath()
    Thread(target=someGuiCallback_2, args=(path,)).start()

def someGuiCallback_2(path):
    md5sum = getMd5Sum(path)
    glib.idle_add(someGuiCallback_3, md5sum)

def someGuiCallback_3(md5sum):
    showNotification("Md5Sum of file: %s" % md5sum)
    ...

(glib.idle_add just pushes a function onto the queue of the main thread)

I've thought about using decoraters, but they don't allow me to access the 'content' of the function after the call. (the showNotification part)

I guess I could write a 'compiler' to change the code before execution, but it doesn't seam like the optimal solution.

Do you have any ideas, on how to do something like the above?

Answer 1

You can use import hooks to achieve this goal...

• PEP 302 - New Import Hooks
• PEP 369 - Post Import Hooks

... but I'd personally view it as a little bit nasty.

If you want to go down that route though, essentially what you'd be doing is this:

• You add an import hook for an extension (eg ".thpy")
• That import hook is then responsible for (essentially) passing some valid code as a result of the import.
• That valid code is given arguments that effectively relate to the file you're importing.
• That means your precompiler can perform whatever transformations you like to the source on the way in.

On the downside:

• Whilst using import hooks in this way will work, it will surprise the life out of any maintainer or your code. (Bad idea IMO)
• The way you do this relies upon imputil - which has been removed in python 3.0, which means your code written this way has a limited lifetime.

Personally I wouldn't go there, but if you do, there's an issue of the Python Magazine where doing this sort of thing is covered in some detail, and I'd advise getting a back issue of that to read up on it. (Written by Paul McGuire, April 2009 issue, probably available as PDF).

Specifically that uses imputil and pyparsing as it's example, but the principle is the same.

Answer 2

How about something like this:

def performAsync(asyncFunc, notifyFunc):
    def threadProc():
        retValue = asyncFunc()
        glib.idle_add(notifyFunc, retValue)
    Thread(target=threadProc).start()

def someGuiCallback(filebutton):
    path = filebutton.getPath()
    performAsync(
        lambda: getMd5Sum(path),
        lambda md5sum: showNotification("Md5Sum of file: %s" % md5sum)
    )

A bit ugly with the lambdas, but it's simple and probably more readable than using precompiler tricks.

Answer 3

Sure you can access function code (already compiled) from decorator, disassemble and hack it. You can even access the source of module it's defined in and recompile it. But I think this is not necessary. Below is an example using decorated generator, where yield statement serves as a delimiter between synchronous and asynchronous parts:

from threading import Thread
import hashlib

def async(gen):
    def func(*args, **kwargs):
        it = gen(*args, **kwargs)
        result = it.next()
        Thread(target=lambda: list(it)).start()
        return result
    return func

@async
def test(text):
    # synchronous part (empty in this example)
    yield # Use "yield value" if you need to return meaningful value
    # asynchronous part[s]
    digest = hashlib.md5(text).hexdigest()
    print digest