Programmatically login to website from Java application

Question

Hello Folks,

I am using Apache's HttpCommons 3.1 package to login to a website.

The website is jsp page with the form using j_security_check along with j_username and j_password.

From a browser, I would do the following:

1. o to the Login.jsp page
2. fill in user name and password
3. click on login button

With the valid username and password, I would get into the restricted resources-sound simple enough; the problem is I have been trying to do this via a simple java application using Apache's HttpCommons 3.1 package (HttpClient, GetMethod, PostMethod etc.) and I have been getting 302-resource moved from the server.

Here is my source code:

String strURL = "http://host:port/app/login/LoginForm.jsp";
        HttpState initialState = new HttpState();
        HttpClient httpclient = new HttpClient();
        httpclient.getHttpConnectionManager().getParams().setConnectionTimeout(30000);
        httpclient.setState(initialState);
        httpclient.getParams().setCookiePolicy(CookiePolicy.RFC_2109);
        GetMethod httpget = new GetMethod(strURL);
        int result = httpclient.executeMethod(httpget);
        System.out.println("Response status code: " + result);
        Cookie[] cookies = httpclient.getState().getCookies();
        String res1 = httpget.getResponseBodyAsString();
        httpget.releaseConnection();

        PostMethod postMethod = new PostMethod("http://host:port/app/j_security_check");
        NameValuePair[] postData = new NameValuePair[2];
        postData[0] = new NameValuePair("j_username", "username");
        postData[1] = new NameValuePair("j_password", "password");
        postMethod.addParameters(postData);
        postMethod.addRequestHeader("Referer",strURL);

        for (int i = 0; i < cookies.length; i++)
        {
            initialState.addCookie(cookies[i]);
        }
        httpclient.setState(initialState);
        try
        {
            httpclient.executeMethod(postMethod);
        }
        catch (HttpException httpe)
        {
            System.err.print("HttpException");
            System.err.println(httpe.getMessage());
            httpe.printStackTrace();
        }
        catch (IOException ioe)
        {
            System.err.print("IOException");
            System.err.println(ioe.getMessage());
            ioe.printStackTrace();
        }
        String res2 = postMethod.getResponseBodyAsString();
        postMethod.releaseConnection();
        System.out.println(res2);
    }
    catch (IOException ex)
    {
        //print out ...
    }

I had tried the same type of an example against the same server using PHP code (CURL) and I do get forwarded to the page I would expect after a successful login.

Any idea why the same is not working in Java?

Thanks, - MK

Answer 1

It's possible that HttpCommons doesn't support 302, try HttpUnit.

Answer 2

You need to call postMethod.setFollowRedirects(true). The 302 is a redirect to the post login page.

Answer 3

Per the documentation on the website (http://hc.apache.org/httpclient-3.x/redirects.html), HttpClient doesn't handle certain redirects by default because the authors feel they're best handled by the user. If you're looking at scraping web sites for certain content, I've used a library called WebHarvest to great success in the past. You can find it on sourceforge : http://web-harvest.sourceforge.net/