Login to site with HttpClient Post

Question

I am trying to make a program that logs into a site and performs some automated activities. I have been using HttpClient 4.0.1, and using this to get started: http://hc.apache.org/httpcomponents-client/primer.html.

On this particular site, the cookies are not set through a "set-cookie" header, but in javascript.

So far, I am unable to achieve the login.

I've tried the following things:

1. add headers manually for all request headers that appear in firebug

   NameValuePair[] data = {
         new BasicNameValuePair("Host",host),
         new BasicNameValuePair("User-Agent"," Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7"),
         new BasicNameValuePair("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"),
         new BasicNameValuePair("Accept-Language","en-us,en;q=0.5"),
         new BasicNameValuePair("Accept-Encoding","gzip,deflate"),
         new BasicNameValuePair("Accept-Charset","ISO-8859-1,utf-8;q=0.7,*;q=0.7"),
         new BasicNameValuePair("Keep-Alive","300"),
         new BasicNameValuePair("Connection","keep-alive"),
         new BasicNameValuePair("Referer",referer),
         new BasicNameValuePair("Cookie",cookiestr)
   };

   for(NameValuePair pair : data){
      loginPost.addHeader(pair.getName(),pair.getValue());
   }

1. creating BasicClientCookies and setting using setCookieStore. unfortunately, i can't figure out how to test if the cookies are actually being sent. also, is there a way to test what other automatic parameters are being sent? (like which browser is being emulated, etc).

The response I'm getting is: HTTP/1.1 417 Expectation Failed

I'm still new to this, so does anyone know off-hand what the problem could be? If not, I'll post more details, code, and the site.

Answer 1

You need WireShark or Fiddler. The first is a network analyser (so you'll see what's going on at a very low level); the second acts as a proxy - less transparent, but higher level.

That way you can look in detail at what happens when you log in with a browser, and what's happening when you try doing the same thing in code.

Answer 2

You can use FireBug's "net' feature to see what is happening when you log in with your browser. This way you should be able to figure out which method generates the cookie value, and how it should be set (which path, name). Use this to set the cookie on HttpClient yourself like:

method.setRequestHeader("Cookie", "special-cookie=value");

Answer 3

I'd echo the comment above - use Wireshark to get a clear view of what is being sent from your client. I've just debugged a similar problem myself with Wireshark. Essential.

If you haven't done so I would suggest studying the examples in http://hc.apache.org/httpcomponents-client/examples.html especially "Form based logon".

I'd avoid setting the Http headers using BasicNameValuePair, HttpClient should give you the basics. Modify further with HttpParams and HttpConnectionParams/HttpProtocolParams. The example conn/ManagerConnectDirect shows how to modify headers.

Answer 4

Here is a snippet that shows how to use post method with params in httpclient 4.