What's the fastest way to check if a web site is up in Perl or C?

Question

Hello all,

I'm trying to check if a web site is up and running. I'm currently doing this with UserAgent library in Perl with timeout 1. However, it is still too slow for me.

I call the script every five minutes from cron. There are lots of links to check and the script takes more than five minutes to complete execution. So, I need a more efficient way to do this. It can even be a solution in C.

Answer 1

curl -I http://hostname

First line will contain 503 or 404 if service not avaliable or page not found.

time yields this for curl -I http://www.google.com

real    0m0.125s
user    0m0.004s
sys     0m0.004s

and this for curl -I http://www.google.cmo

real    0m0.120s
user    0m0.004s
sys  0m0.004s

Answer 2

If there are lots of links, I suggest you make the program multi-threaded or fork() it a few times. That way, you can expect speed improvements.

Answer 3

How about using httping?

Answer 4

Fetching resources from the network usually involves quite a bit of latency.

As Alan Haggai Alavi suggested, you will probably want to divide the work onto several parallel threads/processes. The documentation for the Parallel::ForkManager module even has an example that you should be able to build upon.

Answer 5

It is slow most probably because you're doing it sequentially.

Consider using LWP::Parallel::UserAgent - it will run many requests at the same time.

Answer 6

Following ways to accelerate it:

1. Just check if we can set up a socket to 80 port of target server, do not really send a Get http request, or just send a simple HEAD request.
2. Use multi-thread to make it faster.

Answer 7

I don't know a whole lot of C (BLASPHEMY!) nor Perl, but how I see it I would try the following:

• One thread to do the file writing. This thread would have a que where it could write it's commands in.
• One thread per site you want to check. The thread would use whatever method suits you most of the other answers and then report to the main thread through an event it can trigger.

2cts