Why could Curl be slower than a web browser?

views:

answers:

Why could Curl be slower than a web browser?

I am using this class to make one GET and another POST request to a website (the first request is to set a cookie). I am testing in a Win XP virtual machine with virtualbox, using wamp from wampserver dot com. The 2 requests takes from 10 to 18 seconds (with curl), but if I make those request directly via the webbrowser in that same virtual machine the website loads in just a few seconds, and it retrieves all the images, css, etc.

What could be causing curl to work so slow? is there a way to fix it?

+2 A:

Curl is probably trying to Reverse DNS the server and, as it cant, it just hangs there a little waiting for the timeout.

If the timeout is given by IPV6 you can try CURL_IPRESOLVE_V4 to bypass it altogether. It really depends on your machine configuration and is more a question to Server Fault.

Frankie 2010-09-08 16:41:44

Any way to disable reverse dns? I tried using the ip instead of the domain (the website loads fine using the ip) but curl still takes like 14 seconds on the 2 requests.

jarkam 2010-09-08 16:50:53

You can enter the address directly on the `hosts` file or you can try to force IPv4 and see if it makes any difference.

Frankie 2010-09-08 16:52:08

I used fsockopen and it is taking more than 30mins. do we have to use it like fsockopen('xxx.xxx.xxx.xxx',80) instead of giving the domain name there... and in header Host: hostname.com.. isn't it? anyway let me try frankie's comment.

Jayapal Chandran 2010-09-08 16:53:08

editing the hosts file didn't changed anything

jarkam 2010-09-08 16:56:28

@jarkam but from your testing can you see if it DNS related or not? Follow simple steps and try to eliminate possibilities.

Frankie 2010-09-08 16:57:37

jarkam 2010-09-10 01:13:02

+2 A:

Check your web server logs and try to find any difference between the requests from the normal web browser and the requests from curl

mmonem 2010-09-08 16:41:56

So you mean to check the headers sent by the browser so that using the same headers in curl would make the web server understand something more. Me too had the same problem. i am accessing a server through firefox. it loads faster. but when i used fsockopen('sitename.com') with the same headers like firefox it not load fast. it took around 30 secs or more.

Jayapal Chandran 2010-09-08 16:51:50

@Jayapal Chandran it always takes more and if you time it it will always take around the same time. It's the Reverse DNS timeout and then the real server response (that's not really significant comparing with the timeout).

Frankie 2010-09-08 23:16:26

Oh. actually it was more than 30 seconds like 60. anyway. What i did was i added in the hosts file and i tried the socket_create, socket_connect, socket_read, socket_write functions by specifying the ip. It worked very faster. Here i got confused whether it is because of adding ip in hosts file or directly giving the ip. I checked by removing the ip from hosts and again using fsockopen. this time it worked very fast. This have never happened before. so totally confused. but it is working. What could be the effect?

Jayapal Chandran 2010-09-09 05:50:08

Jayapal, I was using Windows because I don't like installing a webserver on my main OS, and I only had a webserver on a WinXP virtual machine. Now I installed a webserver on a Linux virtual machine (backtrack 4), and curl works like 8x times faster. If you are having this problem on Windows you should try running your scripts on Linux and compare results.

jarkam 2010-09-10 01:11:53

Update: added custom headers to curl emulating exactly what firefox sends and the load time decreased about 50%.

jarkam 2010-09-12 05:11:28

ansaurus

tags:

views:

answers:

Why could Curl be slower than a web browser?

related questions