MPM Prefork Apache Uses Absurd Amount of Memory

Question

Help! My apache processes are all using 115MB of memory on startup.

Relevant information:

Linux version (uname -a)

Linux 2.6.31-14-generic-pae #48-Ubuntu SMP Fri Oct 16 15:22:42 UTC 2009 i686 GNU/Linux

Apache version (/usr/sbin/apache2 -v)

Server version: Apache/2.2.8 (Ubuntu)
Server built:   Mar  9 2010 20:45:36

Top display (top -u www-data)

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
23377 www-data  20   0  115m  94m 3908 S   28  1.6   0:04.59 apache2
23375 www-data  20   0  119m  99m 5892 S    9  1.6   0:05.04 apache2
23324 www-data  20   0  116m  96m 5144 S    2  1.6   0:04.73 apache2
23283 www-data  20   0  115m  95m 4480 S    1  1.6   0:04.89 apache2
23259 www-data  20   0  116m  96m 5380 S    0  1.6   0:05.55 apache2
23370 www-data  20   0  115m  94m 4396 S    0  1.6   0:04.75 apache2
23229 www-data  20   0  116m  96m 6096 S    0  1.6   0:05.43 apache2
... and so on ...

Memory map (pmap $(pidof apache2)) (actually, just one apache2 process)

Most of the memory is [anon], see line 5

23324:   /usr/sbin/apache2 -k start     
08048000    332K r-x--  /usr/sbin/apache2
0809b000      8K rw---  /usr/sbin/apache2
0809d000     12K rw---    [ anon ]       
093a0000  92812K rw---    [ anon ]      
b5b6c000      4K rw---    [ anon ]       
b5b6d000    512K rw-s-    [ shmid=0x13528003 ]
b5fa8000     16K r-x--  /lib/tls/i686/cmov/libnss_dns-2.7.so
b5fac000      8K rw---  /lib/tls/i686/cmov/libnss_dns-2.7.so
b5fae000    120K r-x--  /usr/lib/php5/20060613+lfs/suhosin.so
b5fcc000     16K rw---  /usr/lib/php5/20060613+lfs/suhosin.so
b5fd0000      4K rw---    [ anon ]                           
b5fd1000     76K r-x--  /usr/lib/php5/20060613+lfs/pdo.so    
b5fe4000      8K rw---  /usr/lib/php5/20060613+lfs/pdo.so    
b5fe6000     92K r-x--  /usr/lib/php5/20060613+lfs/mysqli.so 
b5ffd000      8K rw---  /usr/lib/php5/20060613+lfs/mysqli.so 
b5fff000   1648K r-x--  /usr/lib/libmysqlclient.so.15.0.0    
b619b000    268K rw---  /usr/lib/libmysqlclient.so.15.0.0    
b61de000      4K rw---    [ anon ]                           
b61f0000     92K r-x--  /usr/lib/libxcb.so.1.0.0             
b6207000      4K rw---  /usr/lib/libxcb.so.1.0.0             
b6208000    164K r-x--  /usr/lib/libfontconfig.so.1.3.0      
b6231000      4K rw---  /usr/lib/libfontconfig.so.1.3.0      
b6232000    124K r-x--  /usr/lib/libjpeg.so.62.0.0           
b6251000      4K rw---  /usr/lib/libjpeg.so.62.0.0           
b6252000    136K r-x--  /usr/lib/libpng12.so.0.15.0          
b6274000      4K rw---  /usr/lib/libpng12.so.0.15.0          
b6275000     60K r-x--  /usr/lib/libXpm.so.4.11.0            
b6284000      4K rw---  /usr/lib/libXpm.so.4.11.0            
b6285000    912K r-x--  /usr/lib/libX11.so.6.2.0             
b6369000     12K rw---  /usr/lib/libX11.so.6.2.0             
b636c000    424K r-x--  /usr/lib/libfreetype.so.6.3.16       
b63d6000     12K rw---  /usr/lib/libfreetype.so.6.3.16       
b63d9000    236K r-x--  /usr/lib/libt1.so.5.1.1              
b6414000     12K rw---  /usr/lib/libt1.so.5.1.1              
b6417000     84K rw---    [ anon ]                           
b642c000    116K r-x--  /usr/lib/libgd.so.2.0.0              
b6449000    128K rw---  /usr/lib/libgd.so.2.0.0              
b6469000     16K rw---    [ anon ]                           
b646d000     88K r-x--  /usr/lib/php5/20060613+lfs/gd.so     
b6483000     16K rw---  /usr/lib/php5/20060613+lfs/gd.so     
b6487000    192K r-x--  /usr/lib/libidn.so.11.5.30           
b64b7000      4K rw---  /usr/lib/libidn.so.11.5.30           
b64b8000    232K r-x--  /usr/lib/libcurl.so.4.0.1            
b64f2000      4K rw---  /usr/lib/libcurl.so.4.0.1            
b64f8000     44K r-x--  /usr/lib/php5/20060613+lfs/mysql.so  
b6503000      4K rw---  /usr/lib/php5/20060613+lfs/mysql.so  
b6504000    268K r-x--  /usr/lib/libgmp.so.3.4.2             
b6547000      4K rw---  /usr/lib/libgmp.so.3.4.2             
b6548000    648K r-x--  /usr/lib/libclamav.so.5.0.4          
b65ea000     44K rw---  /usr/lib/libclamav.so.5.0.4          
b65f8000     52K r-x--  /usr/lib/php5/20060613+lfs/curl.so   
b6605000      4K rw---  /usr/lib/php5/20060613+lfs/curl.so   
b6606000    148K r-x--  /usr/lib/libmcrypt.so.4.4.7          
b662b000      8K rw---  /usr/lib/libmcrypt.so.4.4.7          
b662d000     28K rw---    [ anon ]                           
b6634000     24K r-x--  /usr/lib/php5/20060613+lfs/pdo_mysql.so
b663a000      4K rw---  /usr/lib/php5/20060613+lfs/pdo_mysql.so
b663b000     16K r-x--  /usr/lib/libXdmcp.so.6.0.0             
b663f000      4K rw---  /usr/lib/libXdmcp.so.6.0.0             
b6640000     12K r-x--  /usr/lib/php5/20060613+lfs/clamav.so   
b6643000      4K rw---  /usr/lib/php5/20060613+lfs/clamav.so   
b6644000   1036K r-x--  /usr/lib/libc-client.so.2007.0         
b6747000     28K rw---  /usr/lib/libc-client.so.2007.0         
b674e000      4K rw---    [ anon ]                             
b6750000     24K r-x--  /usr/lib/libltdl.so.3.1.6              
b6756000      4K rw---  /usr/lib/libltdl.so.3.1.6              
b6757000     32K r-x--  /usr/lib/php5/20060613+lfs/mcrypt.so   
b675f000      4K rw---  /usr/lib/php5/20060613+lfs/mcrypt.so   
b6760000     88K r-x--  /usr/lib/php5/20060613+lfs/imap.so     
b6776000      4K rw---  /usr/lib/php5/20060613+lfs/imap.so     
b6777000    104K r-x--  /usr/local/lib/libssh2.so              
b6791000      4K rw---  /usr/local/lib/libssh2.so              
b6792000   1324K r-x--  /usr/lib/ZendOptimizer.so              
b68dd000     68K rw---  /usr/lib/ZendOptimizer.so              
b68ee000     20K rw---    [ anon ]                             
b68f3000      8K r-x--  /usr/lib/libXau.so.6.0.0               
b68f5000      4K rw---  /usr/lib/libXau.so.6.0.0               
b68f6000     52K r-x--  /usr/lib/php5/20060613+lfs/ssh2.so     
b6903000      4K rw---  /usr/lib/php5/20060613+lfs/ssh2.so     
b6904000    252K r----  /usr/lib/locale/en_US.utf8/LC_CTYPE    
b6974000     64K rw-s-  /dev/zero (deleted)                    
b6984000     36K r-x--  /lib/tls/i686/cmov/libnss_files-2.7.so 
b698d000      8K rw---  /lib/tls/i686/cmov/libnss_files-2.7.so 
b698f000     32K r-x--  /lib/tls/i686/cmov/libnss_nis-2.7.so   
b6997000      8K rw---  /lib/tls/i686/cmov/libnss_nis-2.7.so   
b6999000     28K r-x--  /lib/tls/i686/cmov/libnss_compat-2.7.so
b69a0000      8K rw---  /lib/tls/i686/cmov/libnss_compat-2.7.so
b69a2000     36K r-x--  /lib/libpam.so.0.81.6                  
b69ab000      4K rw---  /lib/libpam.so.0.81.6                  
b69ac000     28K r--s-  /usr/lib/gconv/gconv-modules.cache     
b69b3000      8K r-x--  /usr/lib/apache2/modules/mod_userdir.so
b69b5000      4K rw---  /usr/lib/apache2/modules/mod_userdir.so
b69b6000    148K r-x--  /usr/lib/apache2/modules/mod_ssl.so    
b69db000      8K rw---  /usr/lib/apache2/modules/mod_ssl.so    
b69dd000      8K rw---    [ anon ]                             
b69df000      8K r-x--  /usr/lib/apache2/modules/mod_setenvif.so
b69e1000      4K rw---  /usr/lib/apache2/modules/mod_setenvif.so
b69e2000   1128K r-x--  /usr/lib/libxml2.so.2.6.31              
b6afc000     20K rw---  /usr/lib/libxml2.so.2.6.31              
b6b01000      4K rw---    [ anon ]                              
b6b02000     80K r-x--  /lib/tls/i686/cmov/libnsl-2.7.so        
b6b16000      8K rw---  /lib/tls/i686/cmov/libnsl-2.7.so        
b6b18000      8K rw---    [ anon ]                              
b6b1a000    140K r-x--  /lib/tls/i686/cmov/libm-2.7.so          
b6b3d000      8K rw---  /lib/tls/i686/cmov/libm-2.7.so          
b6b3f000     60K r-x--  /lib/libbz2.so.1.0.4                    
b6b4e000      4K rw---  /lib/libbz2.so.1.0.4                    
b6b4f000      4K r-x--  /usr/lib/libxcb-xlib.so.0.0.0           
b6b50000      4K rw---  /usr/lib/libxcb-xlib.so.0.0.0           
b6b51000     56K r-x--  /usr/lib/apache2/modules/mod_rewrite.so 
b6b5f000      4K rw---  /usr/lib/apache2/modules/mod_rewrite.so 
b6b60000   5060K r-x--  /usr/lib/apache2/modules/libphp5.so     
b7051000    208K rw---  /usr/lib/apache2/modules/libphp5.so     
b7085000     20K rw---    [ anon ]                              
b708a000     28K r-x--  /usr/lib/apache2/modules/mod_negotiation.so
b7091000      4K rw---  /usr/lib/apache2/modules/mod_negotiation.so
b7092000     12K r-x--  /usr/lib/apache2/modules/mod_mime.so       
b7095000      4K rw---  /usr/lib/apache2/modules/mod_mime.so       
b7096000     36K r-x--  /usr/lib/apache2/modules/mod_include.so    
b709f000      4K rw---  /usr/lib/apache2/modules/mod_include.so    
b70a0000      4K r-x--  /usr/lib/apache2/modules/mod_env.so        
b70a1000      4K rw---  /usr/lib/apache2/modules/mod_env.so        
b70a2000      4K r-x--  /usr/lib/apache2/modules/mod_dir.so        
b70a3000      4K rw---  /usr/lib/apache2/modules/mod_dir.so        
b70a4000     20K r-x--  /usr/lib/apache2/modules/mod_cgi.so        
b70a9000      4K rw---  /usr/lib/apache2/modules/mod_cgi.so        
b70aa000     28K r-x--  /usr/lib/apache2/modules/mod_autoindex.so  
b70b1000      4K rw---  /usr/lib/apache2/modules/mod_autoindex.so  
b70b2000      4K r-x--  /usr/lib/apache2/modules/mod_authz_user.so 
b70b3000      4K rw---  /usr/lib/apache2/modules/mod_authz_user.so 
b70b4000      8K r-x--  /usr/lib/apache2/modules/mod_authz_host.so 
b70b6000      4K rw---  /usr/lib/apache2/modules/mod_authz_host.so 
b70b7000      8K r-x--  /usr/lib/apache2/modules/mod_authz_groupfile.so
b70b9000      4K rw---  /usr/lib/apache2/modules/mod_authz_groupfile.so
b70ba000      8K rw---    [ anon ]                                     
b70bc000     12K r-x--  /lib/libgpg-error.so.0.3.0                     
b70bf000      4K rw---  /lib/libgpg-error.so.0.3.0                     
b70c0000      4K rw---    [ anon ]                                     
b70c1000      8K r-x--  /lib/libkeyutils-1.2.so                        
b70c3000      4K rw---  /lib/libkeyutils-1.2.so                        
b70c4000     28K r-x--  /usr/lib/libkrb5support.so.0.1                 
b70cb000      4K rw---  /usr/lib/libkrb5support.so.0.1                 
b70cc000    136K r-x--  /usr/lib/libk5crypto.so.3.1                    
b70ee000      4K rw---  /usr/lib/libk5crypto.so.3.1                    
b70ef000    300K r-x--  /lib/libgcrypt.so.11.2.3                       
b713a000      8K rw---  /lib/libgcrypt.so.11.2.3                       
b713c000     80K r-x--  /usr/lib/libz.so.1.2.3.3                       
b7150000      4K rw---  /usr/lib/libz.so.1.2.3.3                       
b7151000      4K rw---    [ anon ]                                     
b7152000     60K r-x--  /usr/lib/libtasn1.so.3.0.12                    
b7161000      4K rw---  /usr/lib/libtasn1.so.3.0.12                    
b7162000    160K r-x--  /usr/lib/libgssapi_krb5.so.2.2                 
b718a000      4K rw---  /usr/lib/libgssapi_krb5.so.2.2                 
b718b000      8K r-x--  /lib/libcom_err.so.2.1                         
b718d000      4K rw---  /lib/libcom_err.so.2.1                         
b718e000    556K r-x--  /usr/lib/libkrb5.so.3.3                        
b7219000      8K rw---  /usr/lib/libkrb5.so.3.3                        
b721b000   1192K r-x--  /usr/lib/i686/cmov/libcrypto.so.0.9.8          
b7345000     84K rw---  /usr/lib/i686/cmov/libcrypto.so.0.9.8          
b735a000     16K rw---    [ anon ]                                     
b735e000    248K r-x--  /usr/lib/i686/cmov/libssl.so.0.9.8             
b739c000     16K rw---  /usr/lib/i686/cmov/libssl.so.0.9.8             
b73a0000    452K r-x--  /usr/lib/libgnutls.so.13.9.1                   
b7411000     20K rw---  /usr/lib/libgnutls.so.13.9.1                   
b7416000     88K r-x--  /usr/lib/libsasl2.so.2.0.22                    
b742c000      4K rw---  /usr/lib/libsasl2.so.2.0.22                    
b742d000     60K r-x--  /lib/tls/i686/cmov/libresolv-2.7.so            
b743c000      8K rw---  /lib/tls/i686/cmov/libresolv-2.7.so            
b743e000      8K rw---    [ anon ]                                     
b7440000      8K r-x--  /lib/tls/i686/cmov/libdl-2.7.so                
b7442000      8K rw---  /lib/tls/i686/cmov/libdl-2.7.so                
b7444000     36K r-x--  /lib/tls/i686/cmov/libcrypt-2.7.so             
b744d000      8K rw---  /lib/tls/i686/cmov/libcrypt-2.7.so             
b744f000    160K rw---    [ anon ]                                     
b7477000     28K r-x--  /lib/tls/i686/cmov/librt-2.7.so                
b747e000      8K rw---  /lib/tls/i686/cmov/librt-2.7.so                
b7480000     12K r-x--  /lib/libuuid.so.1.2                            
b7483000      4K rw---  /lib/libuuid.so.1.2                            
b7484000    124K r-x--  /usr/lib/libexpat.so.1.5.2                     
b74a3000      8K rw---  /usr/lib/libexpat.so.1.5.2                     
b74a5000    396K r-x--  /usr/lib/libsqlite3.so.0.8.6                   
b7508000      8K rw---  /usr/lib/libsqlite3.so.0.8.6                   
b750a000    120K r-x--  /usr/lib/libpq.so.5.1                          
b7528000      4K rw---  /usr/lib/libpq.so.5.1                          
b7529000   1172K r-x--  /usr/lib/libdb-4.6.so                          
b764e000      8K rw---  /usr/lib/libdb-4.6.so                          
b7650000      4K rw---    [ anon ]                                     
b7651000     48K r-x--  /usr/lib/liblber-2.4.so.2.0.5                  
b765d000      4K rw---  /usr/lib/liblber-2.4.so.2.0.5                  
b765e000    244K r-x--  /usr/lib/libldap_r-2.4.so.2.0.5                
b769b000      4K rw---  /usr/lib/libldap_r-2.4.so.2.0.5                
b769c000      8K rw---    [ anon ]                                     
b769e000   1316K r-x--  /lib/tls/i686/cmov/libc-2.7.so                 
b77e7000      4K r----  /lib/tls/i686/cmov/libc-2.7.so                 
b77e8000      8K rw---  /lib/tls/i686/cmov/libc-2.7.so                 
b77ea000     12K rw---    [ anon ]                                     
b77ed000     80K r-x--  /lib/tls/i686/cmov/libpthread-2.7.so           
b7801000      8K rw---  /lib/tls/i686/cmov/libpthread-2.7.so           
b7803000      8K rw---    [ anon ]                                     
b7805000    136K r-x--  /usr/lib/libapr-1.so.0.2.11                    
b7827000      4K rw---  /usr/lib/libapr-1.so.0.2.11                    
b7828000      4K rw---    [ anon ]
b7829000    100K r-x--  /usr/lib/libaprutil-1.so.0.2.11
b7842000      4K rw---  /usr/lib/libaprutil-1.so.0.2.11
b7843000    152K r-x--  /usr/lib/libpcre.so.3.12.1
b7869000      4K rw---  /usr/lib/libpcre.so.3.12.1
b786a000      4K r-x--  /usr/lib/apache2/modules/mod_authz_default.so
b786b000      4K rw---  /usr/lib/apache2/modules/mod_authz_default.so
b786c000      4K r-x--  /usr/lib/apache2/modules/mod_authn_file.so
b786d000      4K rw---  /usr/lib/apache2/modules/mod_authn_file.so
b786e000     24K r-x--  /usr/lib/apache2/modules/mod_auth_digest.so
b7874000      4K rw---  /usr/lib/apache2/modules/mod_auth_digest.so
b7875000      8K r-x--  /usr/lib/apache2/modules/mod_auth_basic.so
b7877000      4K rw---  /usr/lib/apache2/modules/mod_auth_basic.so
b7878000      8K r-x--  /usr/lib/apache2/modules/mod_alias.so
b787a000      4K rw---  /usr/lib/apache2/modules/mod_alias.so
b787b000      8K rw---    [ anon ]
b787d000      4K r-x--    [ anon ]
b787e000    104K r-x--  /lib/ld-2.7.so
b7898000      8K rw---  /lib/ld-2.7.so
bfd68000     76K rwx--    [ stack ]
bfd7b000      8K rw---    [ anon ]
 total   119008K

I have no idea what's going on. I've tried adjusting the usual parameters (MaxClients, MaxRequestsPerClient, etc, but those don't do anything.) Note, also, that this is memory usage on startup - it doesn't grow, it just starts like this and then stays more or less constant.

Here are values:

Timeout 300
KeepAlive On
MaxKeepAliveRequests 500
KeepAliveTimeout 5
StartServers 1
MinSpareServers 1
MaxSpareServers 5
MaxClients 50
MaxRequestsPerChild 0

We also have the following modules loaded (not compiled in):

alias auth_basic auth_digest authn_file authz_default authz_groupfile authz_host authz_user autoindex cgi dir env include mime negotiation php5 rewrite setenvif ssl userdir 

Ideas?

Answer 1

I'm having the exact same problem.
Like you, none of the changes I made to the conf appeared to have any effect.
And like you, this is after restart.

Any progress ?

Answer 2

It turned out that the problem was actually not Apache's fault, but ClamAV's. (The virus scanner). It was loading it's database of viruses into memory everywhere, instead of where it was needed.

It's kind of a specific case, but check to make sure ClamAV isn't loading an 80-Mb database of viruses. If you need ClamAV, use dlopen and the like to load it from PHP, instead of having Apache load it.