Which files are taking most of my process's I/O time?

Question

I have a fairly large python program that is causing a lot of disk I/O (on top, %wa can get as high as 80, and iotop says that my process is the culprit).

There are several things that may cause this - I'm writing to more than one log file, and I'm saving cached results to disk in several places, so it's not immediately obvious where I should focus my attention.

Is there a linux tool or a python trick that will allow me to see which subsystem is causing the most I/O operations?

Answer 1

You should have a look at SystemTap. It's very powerful tracing and profiling mechanism for Linux system calls: http://sourceware.org/systemtap/wiki

I'm sure it is possible to trace exactly which file descriptor is responsible for IO load - but it will get complicated to start with systemtap.