Redirect with Curl/HttpClient: header "Location" missing.

Question

Hi all, when I get the following url with curl

curl -D headers.http "http://www.springerlink.com/index/10.1007/s00453-007-9157-8"

the file headers.http contains a "Location" header:

HTTP/1.1 302 Found
Date: Tue, 27 Oct 2009 17:00:20 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Location: http://www.springerlink.com/link.asp?id=c104731297q64224
Set-Cookie: CookiesSupported=True; expires=Wed, 27-Oct-2010 17:00:20 GMT; path=/
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 173

but when I used the apache httpclient library this "Location:" header was missing (?).

int status = httpClient.executeMethod(method);
if(status!=HttpStatus.SC_OK &&
status!=HttpStatus.SC_MOVED_TEMPORARILY &&
status!=HttpStatus.SC_MOVED_PERMANENTLY
    )
    {
    throw new IOException("connection failure for "+url+" status:"+status);
    }
Header header=method.getResponseHeader("Location");
if(header==null )
    {

    for(Header h:method.getResponseHeaders())
     {
     LOG.info(h.toString());
     }

    throw new IOException(
     "Expected a redirect for "+url
     );
    }

I've listed the headers below:

INFO: Date: Tue, 27 Oct 2009 17:05:13 GMT
INFO: Server: Microsoft-IIS/6.0
INFO: X-Powered-By: ASP.NET
INFO: X-AspNet-Version: 2.0.50727
INFO: Set-Cookie: ASP.NET_SessionId=js1o5wqnuhuh24islnvkyr45; path=/; HttpOnly
INFO: Cache-Control: private
INFO: Content-Type: text/html; charset=utf-8
INFO: Content-Length: 17245

uhh ???

Answer 1

What's going on is that with curl , you are getting a 302 which is actually a redirect, to the URL in the location header.

With the Apache httpclient it is doing the redirect for you, and returning the headers from the request to the redirected-to location.

To demonstrate this try

curl -D headers.http "http://www.springerlink.com/link.asp?id=c104731297q64224"

and compare the response.

edit: There are actually about 4 redirects in there if you follow each location header through with curl.

Answer 2

http://www.springerlink.com/index/10.1007/s00453-007-9157-8 is actually a redirect. Since the -D option means "headers only", the first one is not redirecting to the specified Location: ..., while the second one is. Take a look at the Content-Length, it's much different.

What happens when you leave out the -D?

Answer 3

Add this

  method.setFollowRedirects(false);

Before you execute the method.

HttpClient follows the redirect automatically by default but Curl doesn't.