Processing a website by using POST data and cookies

Question

I try to access an ASPX-website where subsequent pages are returned based on post data. Unfortunately all my attempts to get the following pages fail. Hopefully, someone here has an idea where to find the error!

In step one I read the session ID from the cookie as well as the value of the viewstate variable in the returned html page. Step two intends to send it back to the server to get the desired page.

Sniffing the data in the webbrowser gives

Host=www.geocaching.com
User-Agent=Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100618
Iceweasel/3.5.9 (like Firefox/3.5.9)
Accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language=en-us,en;q=0.5
Accept-Encoding=gzip,deflate
Accept-Charset=ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive=300
Connection=keep-alive
Referer=http://www.geocaching.com/seek/nearest.aspx?state_id=149
Cookie=Send2GPS=garmin; BMItemsPerPage=200; maprefreshlock=true; ASP.
NET_SessionId=c4jgygfvu1e4ft55dqjapj45
Content-Type=application/x-www-form-urlencoded
Content-Length=4099
POSTDATA=__EVENTTARGET=ctl00%24ContentBody%24pgrBottom%
24lbGoToPage_3&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPD[...]2Xg%3D%
3D&language=on&logcount=on&gpx=on

Currently, my script looks like this

import java.net.*;
import java.io.*;
import java.util.*;
import java.security.*;
import java.net.*;

public class test1 {
    public static void main(String args[]) {
        // String loginWebsite="http://geocaching.com/login/default.aspx";
        final String loginWebsite = "http://www.geocaching.com/seek/nearest.aspx?state_id=159";
        final String POST_CONTENT_TYPE = "application/x-www-form-urlencoded";

        // step 1: get session ID from cookie
        String sessionId = "";
        String viewstate = "";
        try {
            URL url = new URL(loginWebsite);

            String key = "";
            URLConnection urlConnection = url.openConnection();

            if (urlConnection != null) {
                for (int i = 1; (key = urlConnection.getHeaderFieldKey(i)) != null; i++) {
                    // get ASP.NET_SessionId from cookie
                    // System.out.println(urlConnection.getHeaderField(key));
                    if (key.equalsIgnoreCase("set-cookie")) {
                        sessionId = urlConnection.getHeaderField(key);
                        sessionId = sessionId.substring(0, sessionId.indexOf(";"));

                    }
                }

                BufferedReader in = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));

                // get the viewstate parameter
                String aLine;
                while ((aLine = in.readLine()) != null) {
                    // System.out.println(aLine);
                    if (aLine.lastIndexOf("id=\"__VIEWSTATE\"") > 0) {
                        viewstate = aLine.substring(aLine.lastIndexOf("value=\"") + 7, aLine.lastIndexOf("\" "));
                    }
                }
            }
        } catch (IOException e) {
            e.printStackTrace();
        }

        System.out.println(sessionId);
        System.out.println("\n");
        System.out.println(viewstate);
        System.out.println("\n");

        // String goToPage="3";

        // step2: post data to site
        StringBuilder htmlResult = new StringBuilder();
        try {

            String encoded = "__EVENTTARGET=ctl00$ContentBody$pgrBottom$lbGoToPage_3" + "&" + "__EVENTARGUMENT=" + "&"
                + "__VIEWSTATE=" + viewstate;

            URL url = new URL(loginWebsite);
            URLConnection urlConnection = url.openConnection();
            urlConnection = url.openConnection();

            // Specifying that we intend to use this connection for input
            urlConnection.setDoInput(true);

            // Specifying that we intend to use this connection for output
            urlConnection.setDoOutput(true);

            // Specifying the content type of our post
            urlConnection.setRequestProperty("Content-Type", POST_CONTENT_TYPE);

            // urlConnection.setRequestMethod("POST");

            urlConnection.setRequestProperty("Cookie", sessionId);
            urlConnection.setRequestProperty("Content-Type", "text/html");

            DataOutputStream out = new DataOutputStream(urlConnection.getOutputStream());
            out.writeBytes(encoded);
            out.flush();
            out.close();

            BufferedReader in = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));

            String aLine;
            while ((aLine = in.readLine()) != null) {
                System.out.println(aLine);
            }

        } catch (MalformedURLException e) {
            // Print out the exception that occurred
            System.err.println("Invalid URL " + e.getMessage());
        } catch (IOException e) {
            // Print out the exception that occurred
            System.err.println("Unable to execute " + e.getMessage());
        }
    }
}

Any idea what's wrong? Any help is very appreciated!

Update

Thank you for the fast reply!

I switched to use the HttpURLConnection instead of the URLConnection which implements the setRequestMethod(). I also corrected the minor mistakes you mentioned, e.g. removed the obsolete first setRequestProperty call.

Unfortunately this doesn’t change anything... I think I set all relevant parameters but still get the first page of the list, only. It seems that the "__EVENTTARGET=ctl00$ContentBody$pgrBottom$lbGoToPage_3" is ignored. I don't have any clues why.

Internally, the form on the website looks like this:

<form name="aspnetForm" method="post" action="nearest.aspx?state_id=159" id="aspnetForm">

It is called by the following javascript:

<script type="text/javascript"> 
//<![CDATA[
var theForm = document.forms['aspnetForm'];
if (!theForm) {
    theForm = document.aspnetForm;
}
function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}
//]]>
</script> 

Hopefully, this helps to find a solution?

Greetings maik.

Answer 1

Do you actually want to GET or POST? If you want to POST, then you may need the setRequestMethd() line.

You're setting Content-Type twice -- I think you may need to combine these into one line.

Then, don't close the output stream before you try and read from the input stream.

Other than that, is there any more logging you can put in/clues you can give as to what way it's going wrong in?