asynchronous programming in python

Question

Is there a generic notion of asynchronous programming in python? Could I assign a callback to a function, execute it and return to the main program flow immediately, no matter how long the execution of that function would take?

Answer 1

Take a look here:

Asynchronous Programming in Python

An Introduction to Asynchronous Programming and Twisted

Answer 2

You may well want to checkout the Twisted library for Python. They provide many useful tools.

1. A little primer
2. Defer and Related stuff

Answer 3

What you describe (the main program flow resuming immediately while another function executes) is not what's normally called "asynchronous" (AKA "event-driven") programming, but rather "multitasking" (AKA "multithreading" or "multiprocessing"). You can get what you described with the standard library modules threading and multiprocessing (the latter allows actual concurrent execution on multi-core machines).

Asynchronous (event-driven) programming is supported in the standard Python library in the asyncore and asynchat modules, which are very oriented to networking tasks (indeed they internally use the select module, which, on Windows, only supports sockets -- though on Unixy OSs it can also support any file descriptor).

For a more general (though also mostly networking oriented, but not limited to that) support for asynchronous (event-driven) programming, check out the twisted third-party package.

Answer 4

The other respondents are pointing you to Twisted, which is a great and very comprehensive framework but in my opinion it has a very un-pythonic design. Also, AFAICT, you have to use the Twisted main loop, which may be a problem for you if you're already using something else that provides its own loop.

Here is a contrived example that would demonstrate using the threading module:

from threading import Thread

def background_stuff():
  while True:
    print "I am doing some stuff"

t = Thread(target=background_stuff)
t.run()

# Continue doing some other stuff now

However, in pretty much every useful case, you will want to communicate between threads. You should look into synchronization primitives, and become familiar with the concept of concurrency and the related issues.

The threading module provides many such primitives for you to use, if you know how to use them.

Answer 5

You may see my Python Asynchronous Programming tool: http://www.ideawu.com/blog/5.html

import time, random, sys
from delegate import *

def proc(a):
    time.sleep(random.random())
    return str(a)

def proc_callback(handle, args=None):
    ret = d.end(handle)

d = Delegate()
d.init(2) # number of workers

handle = d.begin(proc, '12345', proc_callback, 'test')
sys.stdin.readline()

d.free()