I run a server with the following specs:
- Intel i7 920
- 8 GB RAM
- Linux 2.6.32-25-server #44-Ubuntu 10.04 SMP Fri Sep 17 21:13:39 UTC 2010 x86_64 GNU/Linux
- 75 Apache processes
- Low-end hardware RAID-1 with 2 disks
Historically all our problems with scaling the service have been disk bound but currently we see higher load numbers than before, especially after updating to Ubuntu 10.04. The server handles around 50 requests per second. Swap is not used, and should not be active. The MySQL dataset is some gigabytes but access should be fairly good optimized.
> top
top - 10:42:50 up 16 days, 18:49, 1 user, load average: 20.02, 16.17, 11.44
Tasks: 277 total, 4 running, 273 sleeping, 0 stopped, 0 zombie
Cpu0 : 38.6%us, 3.3%sy, 0.0%ni, 58.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 37.9%us, 3.3%sy, 0.0%ni, 58.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 25.9%us, 3.0%sy, 0.0%ni, 69.5%id, 1.3%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu3 : 23.5%us, 2.0%sy, 0.0%ni, 67.9%id, 0.0%wa, 0.0%hi, 6.6%si, 0.0%st
Cpu4 : 16.4%us, 1.3%sy, 0.0%ni, 82.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 15.3%us, 1.3%sy, 0.0%ni, 83.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 14.3%us, 1.0%sy, 0.0%ni, 84.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 2.3%us, 0.6%sy, 0.0%ni, 97.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8187668k total, 8117276k used, 70392k free, 178920k buffers
Swap: 4198968k total, 2084k used, 4196884k free, 6159328k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32216 mysql 20 0 2026m 788m 4132 S 41 9.9 1292:40 mysqld
8104 www-data 20 0 491m 106m 95m S 4 1.3 1:57.62 apache2
27072 www-data 20 0 684m 112m 101m S 4 1.4 2:51.47 apache2
3391 www-data 20 0 683m 109m 98m S 4 1.4 2:22.29 apache2
16822 www-data 20 0 682m 114m 104m S 4 1.4 3:33.05 apache2
27068 www-data 20 0 555m 113m 102m S 4 1.4 2:53.77 apache2
27118 www-data 20 0 683m 119m 106m S 4 1.5 4:41.48 apache2
1036 www-data 20 0 685m 112m 100m S 3 1.4 2:27.24 apache2
3503 www-data 20 0 556m 81m 70m S 3 1.0 0:33.77 apache2
29803 www-data 20 0 682m 111m 101m S 3 1.4 2:47.09 apache2
1345 www-data 20 0 491m 115m 104m S 3 1.4 4:04.62 apache2
3001 www-data 20 0 379m 109m 98m S 3 1.4 2:13.36 apache2
[... 75 Apache processes with similar specs, but less CPU]
My question is - do you generally see any problems with the high load numbers? The resoponstime has increased, but only by ~30%. Do the load numbers include disk activity to some extent? Do you have any comments what I should focus on during optimizing? Thank you very much!
> iotop
Total DISK READ: 179.70 K/s | Total DISK WRITE: 1735.81 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
16512 be/4 mysql 0.00 B/s 22.94 K/s ?unavailable? mysqld
20701 be/4 mysql 0.00 B/s 0.00 B/s ?unavailable? mysqld
21556 be/4 mysql 0.00 B/s 22.94 K/s ?unavailable? mysqld
28998 be/4 www-data 0.00 B/s 3.82 K/s ?unavailable? apache2 -k start
12771 be/4 mysql 0.00 B/s 3.82 K/s ?unavailable? mysqld
16824 be/4 www-data 0.00 B/s 3.82 K/s ?unavailable? apache2 -k start
2700 be/4 mysql 0.00 B/s 7.65 K/s ?unavailable? mysqld
3074 be/4 mysql 22.94 K/s 0.00 B/s ?unavailable? mysqld
17585 be/4 mysql 0.00 B/s 15.29 K/s ?unavailable? mysqld
30723 be/4 mysql 7.65 K/s 0.00 B/s ?unavailable? mysqld
29906 be/4 www-data 0.00 B/s 3.82 K/s ?unavailable? apache2 -k start
29907 be/4 mysql 0.00 B/s 15.29 K/s ?unavailable? mysqld
13547 be/4 www-data 0.00 B/s 3.82 K/s ?unavailable? apache2 -k start
7444 be/4 www-data 0.00 B/s 3.82 K/s ?unavailable? apache2 -k start
1944 be/4 mysql 149.11 K/s 0.00 B/s ?unavailable? mysqld
16825 be/4 mysql 0.00 B/s 7.65 K/s ?unavailable? mysqld
32223 be/4 mysql 0.00 B/s 3.82 K/s ?unavailable? mysqld
7801 be/4 www-data 0.00 B/s 3.82 K/s ?unavailable? apache2 -k start
5808 be/4 mysql 0.00 B/s 11.47 K/s ?unavailable? mysqld
8104 be/4 www-data 0.00 B/s 3.82 K/s ?unavailable? apache2 -k start
18890 be/4 www-data 0.00 B/s 0.00 B/s ?unavailable? apache2 -k start
1 be/4 root 0.00 B/s 0.00 B/s ?unavailable? init
2 be/4 root 0.00 B/s 0.00 B/s ?unavailable? [kthreadd]
3 rt/4 root 0.00 B/s 0.00 B/s ?unavailable? [migration/0]