views:

2139

answers:

3

This just started happening three weeks or so ago. The content of my website hasn't changed, it's just a phpBB forum using MySQL as a backend.

Nothing has changed in well over a year but recently, every two days or so, the server just shuts down and cannot be accessed at all, I have to notify my service provider to physically restart the machine.

It seems to be tied to these SIGTERM errors I find in the logs. Problem is I have no idea how to fix these kinds of things or find the root cause as my skills in this area are lacking.

Anyone have any ideas what could be going on?

Apache/2.2.3 (CentOS)

20051115

Linux 2.6.18-028stab057.4 #1 SMP Fri Aug 1 10:47:59 MSD 2008 x86_64

From the error log: [Sun Nov 01 15:18:53 2009] [notice] caught SIGTERM, shutting down

Not entirely sure that's what's causing me my problems but it doesn't sound like a good thing.

I can post more log information if needed

[Sun Nov 01 05:07:17 2009] [notice] Digest: generating secret for digest authentication ...
[Sun Nov 01 05:07:17 2009] [notice] Digest: done
[Sun Nov 01 05:07:19 2009] [notice] mod_python: Creating 4 session mutexes based on 10 max processes and 0 max threads.
[Sun Nov 01 05:07:19 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 05:07:19 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 05:07:19 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 05:07:19 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Sun Nov 01 05:07:19 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Sun Nov 01 05:07:19 2009] [warn] WARNING: Attempt to change ServerLimit ignored during restart
[Sun Nov 01 05:07:19 2009] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Sun Nov 01 15:16:11 2009] [error] [client 127.0.0.1] Invalid method in request get
[Sun Nov 01 15:18:53 2009] [notice] caught SIGTERM, shutting down
[Sun Nov 01 15:19:27 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Sun Nov 01 15:19:27 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Sun Nov 01 15:19:27 2009] [notice] Digest: generating secret for digest authentication ...
[Sun Nov 01 15:19:27 2009] [notice] Digest: done
[Sun Nov 01 15:19:27 2009] [notice] mod_python: Creating 4 session mutexes based on 10 max processes and 0 max threads.
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Sun Nov 01 15:19:27 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Sun Nov 01 15:19:27 2009] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Sun Nov 01 16:42:31 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 16:47:25 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 16:59:22 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:37:42 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:44:54 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:44:55 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:44:56 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 18:14:45 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 18:40:30 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:10:59 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:35:10 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:43:07 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:43:46 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 21:25:39 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 21:26:03 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 21:56:57 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:21:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:16 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Sun Nov 01 22:25:17 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:18 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:20 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:21 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:22 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:46:47 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:52:56 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:56:07 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:56:51 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:56:58 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:00:41 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:09:59 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:23:49 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:27:14 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:36:25 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:54:06 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:12:44 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:12:50 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:13:23 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:39:06 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:52:50 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:53:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:13:38 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:13:39 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:13:40 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:23:01 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:43:47 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:45:51 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:46:27 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:47:25 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:23:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:25:37 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:25:38 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:52:29 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:52:51 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 03:49:29 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 03:49:48 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 04:35:33 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/README
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde2
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde3
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde-3.0.9
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/Horde
[Mon Nov 02 04:46:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 05:01:06 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/README
[Mon Nov 02 05:01:06 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde
[Mon Nov 02 05:01:07 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde2
[Mon Nov 02 05:01:08 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde3
[Mon Nov 02 05:01:08 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde-3.0.9
[Mon Nov 02 05:01:09 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/Horde
[Mon Nov 02 05:01:09 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 05:01:10 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 05:01:11 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 09:17:58 2009] [notice] caught SIGTERM, shutting down
[Mon Nov 02 09:18:58 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Mon Nov 02 09:18:58 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Mon Nov 02 09:18:58 2009] [notice] Digest: generating secret for digest authentication ...
[Mon Nov 02 09:18:58 2009] [notice] Digest: done
[Mon Nov 02 09:18:58 2009] [notice] mod_python: Creating 4 session mutexes based on 10 max processes and 0 max threads.
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Mon Nov 02 09:18:58 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Mon Nov 02 09:18:58 2009] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Mon Nov 02 09:32:34 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
+1  A: 

Have you asked your provider to investigate? I assume this is not a dedicated server,

On the face of it, this seems like a security exception and somone is trying to exploit it / or there is a process running at a set time which is causing this, can you think of anything that runs on the server every 2 days? Logging tools?

SIGTERM is the signal sent to a process to request its termination. The symbolic constant for SIGTERM is defined in the header file signal.h. Symbolic signal names are used because signal numbers can vary across platforms, however on the vast majority of systems, SIGTERM is signal #15.

Pino
The server is a VPS and my provider told me they don't offer support for this type of thing. It's all pretty much under my control.Would it help figuring out what his happening if I posted more of the error log?
Tom
It may do, but so far the error log is telling us that somthing with admin privalges told the server to shutdown.
Pino
A: 

SIGTERM is used to restart Apache (provided that it's setup in init to auto-restart): http://httpd.apache.org/docs/2.2/stopping.html

The entries you see in the logs are almost certainly there because your provider used SIGTERM for that purpose. If it's truly crashing, not even serving static content, then that sounds like some sort of a thread/connection exhaustion issue. Perhaps a DoS that holds connections open?

Should definitely be something for your provider to investigate.

kdgregory
That's the thing, my provider is telling me it's out of their hands due to the nature of my hosting solution. I keep telling them nothing has changed on my end since this has started happening but it's falling on deaf ears, they just tell me to look at the logs and hire someone to help which is not what I want to hear. I'll add some log info to my OP.
Tom
Things from the logs that I found interesting: "server reached MaxClients setting" at 22:25:16, and all the access forbidden messages. Take a look at your access log and see what's happening during that time.
kdgregory
Also: note that the SIGTERM happened at 09:17:58, while the previous message in the queue was at 05:01:11 -- this again indicates that the SIGTERM was intentional. I'd again look at the access log to see the previous activty -- I'm guessing it will stop somewhere around 05:01.
kdgregory
Finally: I'm not an Apache expert by any means, but all those error messages on restart would worry me. It seems like your provider isn't an Apache expert either.
kdgregory
Oh so the SIGTERM is most likely the provider shutting down the server then? I neglected to look at the times and you're right. The crash looks like it's happening here at 5:01 and then when I figure out what's happening, I notify my provider who restarts the server at 9:17.
Tom
A: 

I had mysterious SIGTERM shutdowns in our L.A.M.P. server, and it turned out to be an error in a custom PHP module, which was caused by mismatched versions. It was found by looking in the apache access/error logs at the time of malfunction. Don't forget to turn error logging on.

dar7yl
Are these the error logs located in /var/logs/httpd? Do you remember the jist of the error message in the log?
Tom