Who are using all the memory on my production server(apache + mysql + rails) ?

Question

I am running a EC2 small instance as my production server. It has 1.7G memory. I noticed it uses almost all memory. However when I check top output, it looks like that only 30% is actually used. Did I misread the top output?

Here is the top output (sorted by %MEM)

top - 21:33:15 up 141 days,  9:39,  2 users,  load average: 0.00, 0.00, 0.00
Tasks:  81 total,   2 running,  79 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1747660k total,  1733580k used,    14080k free,   224144k buffers
Swap:   917496k total,      132k used,   917364k free,  1144808k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                        
11664 mysql     15   0  794m  83m 5020 S  0.0  4.9   0:17.34 mysqld                                                                                                                                                                         
12845 nobody    25   0 52416  38m 3200 S  0.0  2.3   0:02.10 ruby1.8                                                                                                                                                                        
12847 nobody    16   0 52704  38m 2068 S  0.0  2.2   0:02.08 ruby1.8                                                                                                                                                                        
12023 www-data  15   0 37692  10m 4164 S  0.0  0.6   0:01.28 apache2                                                                                                                                                                        
11979 www-data  15   0 37660  10m 4172 S  0.0  0.6   0:01.24 apache2                                                                                                                                                                        
12020 www-data  15   0 37708  10m 4120 S  0.0  0.6   0:01.17 apache2                                                                                                                                                                        
12263 www-data  15   0 37708  10m 4176 S  0.0  0.6   0:00.83 apache2                                                                                                                                                                        
11989 www-data  15   0 37720  10m 4024 S  0.0  0.6   0:01.28 apache2                                                                                                                                                                        
12014 www-data  15   0 37468  10m 4172 S  0.0  0.6   0:01.17 apache2                                                                                                                                                                        
12021 www-data  15   0 37652  10m 3992 S  0.0  0.6   0:01.25 apache2                                                                                                                                                                        
12054 www-data  15   0 37480  10m 4176 S  0.0  0.6   0:01.33 apache2                                                                                                                                                                        
11990 www-data  15   0 37448  10m 4188 S  0.0  0.6   0:01.16 apache2                                                                                                                                                                        
12024 www-data  16   0 37416  10m 4172 S  0.0  0.6   0:01.00 apache2                                                                                                                                                                        
11991 www-data  15   0 37432  10m 4148 S  0.0  0.6   0:01.24 apache2                                                                                                                                                                        
11984 www-data  15   0 37444 9.8m 3972 S  0.0  0.6   0:01.33 apache2                                                                                                                                                                        
11985 www-data  15   0 37444 9.8m 3948 S  0.0  0.6   0:01.18 apache2                                                                                                                                                                        
11982 www-data  15   0 37408 9.8m 3968 S  0.0  0.6   0:01.12 apache2                                                                                                                                                                        
12013 www-data  17   0 37432 9.8m 4152 S  0.0  0.6   0:01.19 apache2                                                                                                                                                                        
12052 www-data  15   0 37176 9.8m 4180 S  0.0  0.6   0:01.29 apache2                                                                                                                                                                        
11981 www-data  15   0 37172 9.8m 4168 S  0.0  0.6   0:01.40 apache2                                                                                                                                                                        
12395 www-data  15   0 37420 9988 3972 S  0.0  0.6   0:00.72 apache2                                                                                                                                                                        
12015 www-data  15   0 37412 9972 3900 S  0.0  0.6   0:01.31 apache2                                                                                                                                                                        
11987 www-data  15   0 37160 9956 4136 S  0.0  0.6   0:01.22 apache2                                                                                                                                                                        
12022 www-data  15   0 37140 9900 4140 S  0.0  0.6   0:01.20 apache2                                                                                                                                                                        
12051 www-data  15   0 37216 9848 3976 S  0.0  0.6   0:01.31 apache2                                                                                                                                                                        
11978 www-data  18   0 36948 9784 4180 S  0.0  0.6   0:01.08 apache2                                                                                                                                                                        
11975 www-data  15   0 37140 9772 3972 S  0.0  0.6   0:01.49 apache2                                                                                                                                                                        
12019 www-data  15   0 37148 9752 3944 S  0.0  0.6   0:01.08 apache2                                                                                                                                                                        
11970 www-data  15   0 36920 9736 4160 S  0.0  0.6   0:01.25 apache2                                                                                                                                                                        
11974 www-data  15   0 36848 9656 4148 S  0.0  0.6   0:01.53 apache2                                                                                                                                                                        
11973 www-data  15   0 36924 9552 3972 S  0.0  0.5   0:01.19 apache2                                                                                                                                                                        
28622 root      18   0 35232 9232 5592 S  0.0  0.5   0:00.30 apache2                                                                                                                                                                        
11969 www-data  15   0 36340 9132 4136 S  0.0  0.5   0:01.51 apache2                                                                                                                                                                        
12018 www-data  19   0 36332 9124 4136 S  0.0  0.5   0:01.32 apache2                                                                                                                                                                        
11972 www-data  15   0 36320 8968 3988 S  0.0  0.5   0:01.33 apache2                                                                                                                                                                        
12012 www-data  15   0 35796 8600 4144 S  0.0  0.5   0:01.11 apache2                                                                                                                                                                        
11965 root      15   0 17356 7552 1644 S  0.0  0.4   0:00.13 ruby1.8                                                                                                                                                                        
12848 root      15   0  8384 2744 2164 R  0.0  0.2   0:00.12 sshd                                                                                                                                                                           
12762 root      15   0  8384 2724 2164 S  0.0  0.2   0:00.01 sshd                                                                                                                                                                           
11302 postfix   18   0  6184 2576 1880 S  0.0  0.1   0:00.02 tlsmgr                                                                                                                                                                         
11964 root      16   0  8188 2248 1492 S  0.0  0.1   0:00.06 ApplicationPool                                                                                                                                                                
23997 postfix   22   0  5856 1852 1488 S  0.0  0.1   0:00.22 qmgr                                                                                                                                                                           
12850 root      15   0  4408 1848 1436 S  0.0  0.1   0:00.00 bash                                                                                                                                                                           
12764 root      25   0  4396 1800 1400 S  0.0  0.1   0:00.00 bash                                                                                                                                                                           
23996 root      15   0  5804 1780 1428 S  0.0  0.1   0:01.01 master                                                                                                                                                                         
13036 postfix   17   0  5812 1684 1356 S  0.0  0.1   0:00.00 pickup                                                                                                                                                                         
 1051 klog      18   0  2884 1676  436 S  0.0  0.1   0:00.04 klogd                                                                                                                                                                          
13035 root      15   0  2468 1164  916 R  0.0  0.1   0:00.01 top                                                                                                                                                                            
 5841 nobody    15   0  2652 1120  684 S  0.0  0.1   0:00.50 memcached                                                                                                                                                                      
11509 root      15   0  5456 1068  676 S  0.0  0.1   0:00.00 sshd                                                                                                                                                                           
 1163 root      18   0  3560 1060  872 S  0.0  0.1   0:01.46 cron                                                                                                                                                                           
    1 root      18   0  2032  840  580 S  0.0  0.0   0:04.20 init                                                                                                                                                                           
 4070 syslog    18   0  2056  732  568 S  0.0  0.0   7:25.48 syslogd                                                                                                                                                                        
  908 root      16  -2  2292  656  528 S  0.0  0.0   0:00.06 dhclient3                                                                                                                                                                      

Answer 1

Mem: 1747660k total, 1733580k used, 14080k free, 224144k buffers

compare the total and used :-)

Answer 2

It is used for file buffering. It is nothing wrong since good memory managment should always use all availble memory in system. I don't remember but I think that 1144808k cached is the memory you can't find.

You can try writing simple application that reserves about 1 GB of memory and release it and quit. Then probably you should have this 1 GB counted as free memory since file buffers was removed.

Answer 3

The 'used' count includes filesystem cache and kernel buffers. The cached memory can be free'd when an application requires more heap. You are right to say that only about 30% is actually used, since 65% of that is cache, and 12% is buffers.

The kernel will release the cached memory when an application attempts to allocate more memory, this is normal behavior and I see no problem with your memory usage.

When you use significant amounts of swap, and your 'cached' count is very low - then you have a problem.

Some additional helpful information here (applicable to any Linux distro) - http://forums.gentoo.org/viewtopic.php?t=175419