aio_write on linux with rtkaio is sometimes long

Question

I'm using async io on linux with rtkaio library. In my tests everything works perfectly, but, in my real application i see that aio_write which is supposed to return very fast, is very slow. It can take more than 100 milis to write a 128KB to a O_DIRECT padded file. Both my test and the application use same I/O size, i check on the same file system (GFS).

I added counting and i see that there are about 50% of async io operations that are short (shorter then 2 milis) and 50% that are long (longer than 2 milis).

I also checked that the test and the application both use the same rtkaio library.

I'm pretty lost, anyone any ideas where should i look?

Another my related question: http://stackoverflow.com/questions/1799537/proc-sys-fs-aio-nr-is-never-higher-than-1024-aio-on-linux

Answer 1

I don't mean to be discouraging, but I doubt you'll get a very useful answer to this question because there are way too many variables here (a cluster file-system, asynchronous I/O, O_DIRECT, etc.), and unless someone is very familiar with your setup in particular, the best anyone can do is give you vague advice.

I guess you should start by trying to determine whether it's the actual write that's slow, or whether it's the invocation of the async callback handler that's taking long. You can try replacing the calls to aio_write with regular synchronous write calls. Also, the write sizes you mention seem pretty small, so why are you using O_DIRECT? I would think O_DIRECT would decrease performance here, if there are many small writes.

Answer 2

Are you opening the file with open() ?

Ensure that you're opening the file with O_NONBLOCK, and you're opening with O_WRONLY.