Why does my mod_perl script freeze my server?

Question

Hi!

I cannot make my Perl script run stable on the server. Here is the problem.

When the script is accessed more than 5 times a second, the server freezes. And some time later the server hangs forever. SSH does not respond and I have to restart the server.

I'm using Apache with mod_perl.

The script is hosted on Virtual Dedicated Server under Ubuntu. I'm operating it through SSH. These are the server params CPU: 400 MHz RAM: 256 MB

Maximal execution time of the script is 200 milliseconds.

I have monitored server load with the "top" utility. It does not display any problems, this is the CPU statistics during a load of 5 scripts per second:

Cpu(s): 12.1%us,  0.6%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si, 87.2%st

What options do I have to make the script work without problems?

This is the result of ps aux | fgrep perl at the moment of loading:

ps aux | fgrep perl
www-data  2925  0.3  6.5  45520 17064 ?        R    17:00   0:01 /var/www/perl/loa -k start
www-data  2926  0.2  6.5  45520 17068 ?        R    17:00   0:01 /var/www/perl/loa -k start
www-data  2927  0.4  6.5  45676 17060 ?        R    17:00   0:01 /var/www/perl/loa -k start
www-data  2928  0.3  6.5  45676 17060 ?        R    17:00   0:01 /var/www/perl/loa -k start
www-data  2929  0.2  6.5  45676 17060 ?        R    17:00   0:01 /var/www/perl/loa -k start
www-data  2931  0.4  6.5  45740 17076 ?        R    17:00   0:01 /var/www/perl/loa -k start
root      2968  0.0  0.2   3196   656 pts/0    R+   17:06   0:00 fgrep perl

UPDATE

I have found the bottleneck. I've been using DateTime module many times around the code. The following DateTime module methods appear to be very slow.

• new()
• now()
• set(...)
• delta_ms(...)

I'm going to substitute them with fast analogs.

Another concern. mod_perl instance takes a lot of memory. And I have no idea why. I have tried to run a simple perl script that does not import any modules. I run it just after apache restart. The script takes 37M of memory. Why does it happen? Do you know how to force mod_perl do not use the extra memory?

A regular perl script, without mod_perl support, takes 3-5M of memory.

Guys, thank you for so much help, I wasn't expecting such a wonderful response!

UPDATE 2

I have found one more fact. I've created a simple perl script that just waits for 5 seconds.

#!/usr/bin/perl
use CGI;

my $query= new CGI;
my $content = "5 second delay...\n";

$query->header(
    '-Content-type' => "text/plain",
    '-Content-Length' => length($content)
);

print $content;

sleep(5);

Then I spawn many these scripts at the same time. Stealth time (st) in top utility jumps up from 0% to 80% and stays high until the scripts are done.

Where does this load come from?

Also, as I've already mentioned, each perl instance takes 36M of memory.

Answer 1

That's not a very large server. Could it be simply spawning the Perl interpreter that makes it kneel? Loading perl (which I happily assume is more than 1 MB) five times a second might be asking too much.

Of course, it should be cached, but it will still need initialization before being able to execute.

Answer 2

What kind of interface does the script use? You would certainly get a better performance if you could avoid running the perl executable again and again, for example by using FastCGI.

Answer 3

Your numbers from top seem to indicate that other processes outside your VM are throttling your CPU, note the last number, 87.2%st, which indicates that about 87% of your CPU time is being allocated by your hypervisor for tasks outside your VM even though your VM has things it would like to run. Whether this is related to your problem or not is hard to say.

Beyond upgrading your server as suggested by unwind, using a persistent process environment as suggested by zoul, it's possible that your process isn't CPU bound at all but is instead IO-bound, such as to the network or to your disk access, or memory-bound. It's hard to say without more details on what your script is actually doing when it's invoked.

EDIT: Your updated question with info on your memory usage is revealing, as each of your processes wants 45M of ram all to itself, and is sharing 17M more. With just 5 or 6 processes running, you're exceeding the amount of RAM available. That's a good amount of memory for a vanilla Perl script to use, what's it doing with it?

Answer 4

While, by today's standards, the server's specs are not impressive, I have run fairly complicated stuff concurrently on similar hardware. However, I used a very barebones, run only what is necessary FreeBSD configuration. (Similar to what you can achieve using ArchLinux). I suspect you did not do a lot of custom configuration, and accepted Ubuntu defaults which may be too heavy for those specs.

Currently, I am playing around with a Linode 360 and performance is fine.

Now, all this is meant to state the obvious: We need information that you have which you have not shared with us. Web server configuration, memory footprint of the script + interpreter, how many files are open etc etc. Either try to provide the smallest script that still exhibits the problem or provide more information.

Update: Now that I see you are using mod_perl: 1) Have you made sure all libraries needed by the script were preloaded at server start? 2) Are you getting any variable won't stay shared messages in the log? 3) Have you read mod_perl Performance? (Chapter 10: Sharing Memory might be especially relevant).

In general, you should preload common libraries at the start of the Apache server. As a very simplified rule of thumb, the more stuff stays shared, the more you can get out of your server. See Startup File in Practical mod_perl.

Plus, I think 35MB per server is a little much. I think you could cut that down if you eliminated unneeded modules from the Apache configuration. However, even if you could not, say all that 35MB is shared, plus the maximum child process is 50MB, you should be able to accommodate about 20 clients at a time.

I just noticed the script you are testing. Really, try preloading CGI at server startup by adding the following lines to your startup.pl:

use strict;
use warnings;

use CGI();

Second, change that script to

#!/usr/bin/perl

use strict;
use warnings;
use CGI ();

$| = 1;

handle_request();

sub handle_request {
    my $cgi = CGI->new;

    my $content = "5 second delay...\n";

    print $cgi->header('text/plain'), $content;

    sleep(5);
}

Note that you were never sending the header in the original script (I also hate calling a CGI instance $query so I took the liberty of changing that as well). See also Perl Reference.

Report back the memory usage after that.

Finally, why are you sleeping 5 seconds? AFAIK, Apache's default time out for a script is 3 seconds.