How can I speed up CURL tasks?

Question

I'm using CURL to fetch some data from user accounts. First it logs in and then redirect to another URL where the data resides.

My stats showed that it took an average of 14 seconds to fetch some data spread over 5 pages. I would like to speed things up, my questions are:

Is it possible to see how much each step takes? Do you know how could I speed up/enhance CURL?

Thanks

Answer 1

You could try multithreading although I'm too sure whether that would completely work or not.

Answer 2

To make the task /feel/ faster -- Don't run it from web, run it as a periodic task (cron job). Cache the file on disk.

Answer 3

You can't make the process of retrieving a page from a server any faster.

You can make the pages smaller, so they can download quicker. You can beef up the processing power on the servers or the connection between your server and the server the pages are on.

If you are consuming a service, what format is the data in? If it is XML, maybe it is too verbose and this is causing lots of extra kilobytes, for example.

Answer 4

split task to 3 files.

1. file for retrieving page list and as your main script (to put on crontab) (main.php)
2. for parsing the actual page. (parse.php)
3. some shell script to process your 2nd script.

Then, in your 1st file, do something like this:

<?php
$pagelist = get_page_list();//this will retrieve page list using CURL and save each page to some, let's say pagelist.txt and return this absolute path.

exec("/path/to/php /your/3rdscript.sh < $pagelist");
?>

And here's your 3rd file:

#!/bin/bash  

while read line
do
    /path/to/php /path/to/your/2ndscript.php -f $line &
done

Please note that on 3rd script (the shell script) I use & (ampersand). This will tell the shell to put that particular process into background process.

On your 2nd script, you can use something like this:

<?php

$pageurl = $argv[2];
//do your curl process to fetch page $pageurl here

Using step above, you can speed up by fetching several pages at once.

Answer 5

you can use parallelCurl by Pete Warden. The source is available here http://github.com/petewarden/ParallelCurl. The module allows you to run multiple CURL url fetches in parallel in PHP