Slow wget speeds when connecting to https pages

Question

I'm using wget to connect to a secure site like this:

wget -nc -i inputFile

where inputeFile consists of URLs like this:

https://clientWebsite.com/TheirPageName.asp?orderValue=1.00&merchantID=36&programmeID=92&ref=foo&Ofaz=0

This page returns a small gif file. For some reason, this is taking around 2.5 minutes. When I paste the same URL into a browser, I get back a response within seconds.

Does anyone have any idea what could be causing this?

The version of wget, by the way, is "GNU Wget 1.9+cvs-stable (Red Hat modified)"

Answer 1

1. Try forging your UserAgent

   -U "Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-GB; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1"
   

2. Disable Ceritificate Checking ( slow )

   --no-check-certificate
   

3. Debug whats happening by enabling verbostity

   -v
   

4. Eliminate need for DNS lookups:

   Hardcode thier IP address in your HOSTS file

   /etc/hosts 
   123.122.121.120  foo.bar.com
   

Answer 2

Is the certificate on the client site valid? You may want to specify --no-certificate-check if it is a self-signed certificate.

HTTPS (SSL/TLS) Options for wget

Answer 3

Have you tried profiling the requests using strace/dtrace/truss (depending on your platform)?

There are a wide variety of issues that could be causing this. What version of openssl is being used by wget - there could be an issue there. What OS is this running on (full information would be useful there).

There could be some form of download slowdown being enforced due to the agent ID being passed by wget implemented on the site to reduce the effects of spiders.

Is wget performing full certificate validation? Have you tried using --no-check-certificate?

Answer 4

I know this is a year old but this exact problem plagued us for days. Turns out it was our DNS server but I got around it by disabling IP6 on my box.

You can test it out prior to making the system change by adding "--inet4-only" to the end of the command (w/o quotes).