HTTP Builder/Groovy - lost 302 (redirect) handling?

Question

Dear All:

I am reading here http://groovy.codehaus.org/modules/http-builder/doc/handlers.html "In cases where a response sends a redirect status code, this is handled internally by Apache HttpClient, which by default will simply follow the redirect by re-sending the request to the new URL. You do not need to do anything special in order to follow 302 responses."

This seems to work fine when I simply use the get() or post() methods without a closure.

However, when I use a closure, I seem to lose 302 handling. Is there some way I can handle this myself? Thank you

p.s. Here is my log output showing it is a 302 response

 [java] FINER: resp.statusLine: "HTTP/1.1 302 Found"

Here is the relevant code:

// Copyright (C) 2010 Misha Koshelev. All Rights Reserved.
package com.mksoft.fbbday.main

import groovyx.net.http.ContentType

import java.util.logging.Level
import java.util.logging.Logger

class HTTPBuilder {
  def dataDirectory
  HTTPBuilder(dataDirectory) {
    this.dataDirectory=dataDirectory
  }

  // Main logic
  def logger=Logger.getLogger(this.class.name)
  def closure={resp,reader->
    logger.finer("resp.statusLine: \"${resp.statusLine}\"")
    if (logger.isLoggable(Level.FINEST)) {
      def respHeadersString='Headers:';
      resp.headers.each() { header->respHeadersString+="\n\t${header.name}=\"${header.value}\"" }
      logger.finest(respHeadersString)
    }

    def text=reader.text
    def lastHtml=new File("${dataDirectory}${File.separator}last.html")
    if (lastHtml.exists()) {
      lastHtml.delete()
    }
    lastHtml<<text
    new XmlSlurper(new org.cyberneko.html.parsers.SAXParser()).parseText(text)          
  }
  def processArgs(args) {
    if (logger.isLoggable(Level.FINER)) {
      def argsString='Args:';
      args.each() { arg->argsString+="\n\t${arg.key}=\"${arg.value}\"" }
      logger.finer(argsString)
    }
    args.contentType=groovyx.net.http.ContentType.TEXT
    args
  }

  // HTTPBuilder methods
  def httpBuilder=new groovyx.net.http.HTTPBuilder ()
  def get(args) {
    httpBuilder.get(processArgs(args),closure)
  }
  def post(args) {
    args.contentType=groovyx.net.http.ContentType.TEXT
    httpBuilder.post(processArgs(args),closure) 
  }
}

Here is a specific tester:

#!/usr/bin/env groovy

import groovyx.net.http.HTTPBuilder
import groovyx.net.http.Method
import static groovyx.net.http.ContentType.URLENC

import java.util.logging.ConsoleHandler
import java.util.logging.Level
import java.util.logging.Logger

// MUST ENTER VALID FACEBOOK EMAIL AND PASSWORD BELOW !!!
def email=''
def pass=''

// Remove default loggers
def logger=Logger.getLogger('')
def handlers=logger.handlers
handlers.each() { handler->logger.removeHandler(handler) }

// Log ALL to Console
logger.setLevel Level.ALL
def consoleHandler=new ConsoleHandler()
consoleHandler.setLevel Level.ALL
logger.addHandler(consoleHandler)

// Facebook - need to get main page to capture cookies
def http = new HTTPBuilder()
http.get(uri:'http://www.facebook.com')

// Login
def html=http.post(uri:'https://login.facebook.com/login.php?login_attempt=1',body:[email:email,pass:pass])
assert html==null

// Why null?
html=http.post(uri:'https://login.facebook.com/login.php?login_attempt=1',body:[email:email,pass:pass]) { resp,reader->
  assert resp.statusLine.statusCode==302

  // Shouldn't we be redirected???
  // http://groovy.codehaus.org/modules/http-builder/doc/handlers.html
  // "In cases where a response sends a redirect status code, this is handled internally by Apache HttpClient, which by default will simply follow the redirect by re-sending the request to the new URL. You do not need to do anything special in order to follow 302 responses. "
}

Here are relevant logs:

FINE: Receiving response: HTTP/1.1 302 Found
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << HTTP/1.1 302 Found
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Cache-Control: private, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Expires: Sat, 01 Jan 2000 00:00:00 GMT
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Location: http://www.facebook.com/home.php?
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << P3P: CP="DSP LAW"
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Pragma: no-cache
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Set-Cookie: datr=1275687438-9ff6ae60a89d444d0fd9917abf56e085d370277a6e9ed50c1ba79; expires=Sun, 03-Jun-2012 21:37:24 GMT; path=/; domain=.facebook.com
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Set-Cookie: lxe=koshelev%40post.harvard.edu; expires=Tue, 28-Sep-2010 15:24:04 GMT; path=/; domain=.facebook.com; httponly
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Set-Cookie: lxr=deleted; expires=Thu, 04-Jun-2009 21:37:23 GMT; path=/; domain=.facebook.com; httponly
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Set-Cookie: pk=183883c0a9afab1608e95d59164cc7dd; path=/; domain=.facebook.com; httponly
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Content-Type: text/html; charset=utf-8
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << X-Cnection: close
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Date: Fri, 04 Jun 2010 21:37:24 GMT
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Content-Length: 0
Jun 4, 2010 4:37:22 PM org.apache.http.client.protocol.ResponseProcessCookies processCookies
FINE: Cookie accepted: "[version: 0][name: datr][value: 1275687438-9ff6ae60a89d444d0fd9917abf56e085d370277a6e9ed50c1ba79][domain: .facebook.com][path: /][expiry: Sun Jun 03 16:37:24 CDT 2012]". 
Jun 4, 2010 4:37:22 PM org.apache.http.client.protocol.ResponseProcessCookies processCookies
FINE: Cookie accepted: "[version: 0][name: lxe][value: koshelev%40post.harvard.edu][domain: .facebook.com][path: /][expiry: Tue Sep 28 10:24:04 CDT 2010]". 
Jun 4, 2010 4:37:22 PM org.apache.http.client.protocol.ResponseProcessCookies processCookies
FINE: Cookie accepted: "[version: 0][name: lxr][value: deleted][domain: .facebook.com][path: /][expiry: Thu Jun 04 16:37:23 CDT 2009]". 
Jun 4, 2010 4:37:22 PM org.apache.http.client.protocol.ResponseProcessCookies processCookies
FINE: Cookie accepted: "[version: 0][name: pk][value: 183883c0a9afab1608e95d59164cc7dd][domain: .facebook.com][path: /][expiry: null]". 
Jun 4, 2010 4:37:22 PM org.apache.http.impl.client.DefaultRequestDirector execute
FINE: Connection can be kept alive indefinitely
Jun 4, 2010 4:37:22 PM groovyx.net.http.HTTPBuilder doRequest
FINE: Response code: 302; found handler: post302$_run_closure2@7023d08b
Jun 4, 2010 4:37:22 PM groovyx.net.http.HTTPBuilder doRequest
FINEST: response handler result: null
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.SingleClientConnManager releaseConnection
FINE: Releasing connection org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter@605b28c9

You can see there is clearly a location argument.

Thank you Misha

Answer 1

What other headers do you see when you process the 302 response? If you were to turn on http client logging you'd expect to see HttpClient process the 302 response and automatically request URL in the Location header. What do you see when you process that URL? Does it work for any URL?

Try http://www.sun.com (it redirects to Oracle now.) I'm just wondering if the server you're working with is doing something wonky like sending a 302 with no Location header.

Answer 2

I've had the same problem with HTTPBuilder until I realized that the HTTP/1.1 spec states:

Redirection 3xx

[..] This class of status code indicates that further action needs to be
taken by the user agent in order to fulfill the request. The action
required MAY be carried out by the user agent without interaction with the user if and only if the method used in the second request is GET or HEAD.

302 Found

[..] If the 302 status code is received in response to a request other than GET or HEAD, the user agent MUST NOT automatically redirect the request unless it can be confirmed by the user, since this might change the conditions under which the request was issued.

Essentially this means that the request following a POST and 302 redirect won't work automatically and will require user intervention if the HTTP/1.1 spec is followed by the letter. Not all Http clients follow this practice, in fact most browsers do not. However the Apache Http Client (which is the underlying Http client for HttpBuilder) is spec compliant. There is an issue in the Apache Http Client bugtracker that contains more information and a possible solution for the problem.