Debugging 100% iowait problem in linux

Question

I've been trying to trace down why I have 100% iowait on my box. If I do something like a mysql select query, system goes to 100% iowait (on more than one cpu on my server,) which kills my watchdogs and sometimes kills httpd itself.

In vmstat I see that every 8 seconds or so, there's a 5MB disk write. And that causes at least one cpu (out of 4) to be blocking for one or two seconds.

I have to say that there are a few million files in my ext3 (and I tried ext2, and I have no atime and no journaling enabled.) There is a hardware raid, mirroring two 300GB ides.

I'm missing dtrace. Is there any way to find out what causes these writes? and how do I speed my filesystem up?

Ideas are welcome!

Thank you!

Answer 1

OK, possible diagnosis steps (for posterity):

1. Have you confirmed that you're not actually running out of virtual memory and therefore swapping processes out to disk?

2. If it's not the kernel swapping, you may be able to use strace (as you don't have dtrace) to prove whether it's MySQL doing the writes

Can you please provide more details of the hardware and O/S configuration?

Answer 2

Use iotop.

Answer 3

Hi,

im having this problem too..

Im trying to debug a PHP/MySQL app. Im using Netbeans 6.8 and Ubuntu 8.04.

My hardware: Intel Centrino 1,4 GHz, 1,5 GB RAM, 78 % hard disk free.

I don't know how to confirm this:

"Have you confirmed that you're not actually running out of virtual memory and therefore swapping processes out to disk?"

Any hand?

Javi