Asynchronous file writing possible in python?

Question

Is there an existing module or an easy way to use aio_write to write to a file asynchronously in python? I know by file io that comes with Python is all blocking which is fine in most cases, but for this particular case I need writes not to block the application at all, or at least as minimally as possible.

Answer 1

Twisted has non-blocking writes on file descriptors. If you're writing async code, I'd expect you to be using twisted, anyway. :)

Answer 2

As I understand things, asynchronous I/O is not quite the same as non-blocking I/O.

In the case of non-blocking I/O, once a file descriptor is setup to be "non-blocking", a read() system call (for instance) will return EWOULDBLOCK (or EAGAIN) if the read operation would block the calling process in order to complete the operation. The system calls select(), poll(), epoll(), etc. are provided so that the process can ask the OS to be told when one or more file descriptors become available for performing some I/O operation.

Asynchronous I/O operates by queuing a request for I/O to the file descriptor, tracked independently of the calling process. For a file descriptor that supports asynchronous I/O (raw disk devcies typically), a process can call aio_read() (for instance) to request a number of bytes be read from the file descriptor. The system call returns immediately, whether or not the I/O has completed. Some time later, the process then polls the operating system for the completion of the I/O (that is, buffer is filled with data).

A process (single-threaded) that only performs non-blocking I/O will be able to read or write from one file descriptor that is ready for I/O when another is not ready. But the process must still synchronously issue the system calls to perform the I/O to all the ready file descriptors. Whereas, in the asynchronous I/O case, the process is just checking for the completion of the I/O (buffer filled with data). With asynchronous I/O, the OS has the freedom to operate in parallel as much as possible to service the I/O, if it so chooses.

With that, are there any wrappers for the POSIX aio_read/write etc. system calls for Python?

Answer 3

I have answered an similar question here: http://stackoverflow.com/questions/3221314/asynchronous-programming-in-python/3590244#3590244