How can I poll web requests without blocking?

Question

I have two web requests which I need to poll to find out when they return. Ideally I don't want to keep testing them in a tight loop. I would like to free up the CPU so other processes can execute.

I'm currently using Perl's Time::HiRes::sleep(0.100) function to release the CPU before testing whether or not the web requests have returned.

During testing under load I can see that the sleep duration 'stretches'. Ideally I want to make sure that the sleep duration is adhered to but that CPU is freed up. Should I be calling a different function to achieve this?

I'm coding Perl on Linux 2.6.

Answer 1

Rather than polling, see if you can't get file-descriptors and do a select call.

Then you'll get control back as soon as anything happens, without occupying the CPU at all.

Somewhere in the web-request will be some sockets, and attached to the sockets will be file-descriptors that you can use in select.

In any case your program can be interrupted at any point for any amount of time; if this is a real problem you need a real-time operating system, but since you're dealing with web-requests I doubt you need that level of responsiveness.

In fact what you want is a high level interface that does the select call for you. As suggested in the comments: http://search.cpan.org/dist/HTTP-Async/ looks like it'll do precisely what you need.

Answer 2

I don't think sleep duration can be guaranteed on regular Linux. That's pretty much the point of a "Real Time" operating system, and regular Linux is not "Real Time."

I agree with @Douglas Leeder: use a select call to have the kernel notify you when something changes. You can also emulate sub-second sleeps with a select call, but Time::HiRes is a cleaner interface (and you're still not going to avoid stretching the wait).

Answer 3

It sounds like you really want an event loop. There is POE, EV, and abstraction layers over both.

Either way, don't implement this yourself. This wheel has already been invented.