POST doesn't seem to work with my form from HttpURLConnection

Question

I am trying to log into a reports system using java. So far, I have tried MANY different implementations that have used HttpURLConnection. The HTML I am trying to post to boils down to:

<form name="logonForm" method="POST" action="http://remoteserver/logon.object"&gt;
<input type="hidden" name="qryStr" value=""> 
<input type="hidden" name="cmsVisible" value="true"> 
<input type="hidden" name="authenticationVisible" value="true"> 
<input type="hidden" name="referer" value="">
<input type="hidden" name="refererFormData" value="">
<input type="hidden" name="isFromLogonPage" value="true">
<input type="text" name="cms" value="xxxxx" class="textfield" id="apsTextEdit">
<input type="text" name="username" value="xxxxx"class="textfield" id="usernameTextEdit">
<input type="text" name="password" value="xxxxx"class="textfield" id="passwordTextEdit">
<select name="authType"id="authenticationSelectBox">
<option selected value='xxxxx'>xxxxx</select>
<input type="submit" name="SUBMIT">
</form>

I know this form provides valid input, because when I click submit in a browser, it logs me into my reports system. However, I always get the above html back as a response when I send a programmatic request. Note I have tried about a million implementations that have used HttpURLRequest. One example:

        OutputStreamWriter wr = new OutputStreamWriter(connection.getOutputStream());
        wr.write(parameters);
        wr.flush();

        // Get the response
        BufferedReader rd = new BufferedReader(new InputStreamReader(connection.getInputStream()));
        String line;
        String allLines = "";
        while ((line = rd.readLine()) != null) {
            allLines += "\n" + line; // Process line...
        }
        wr.close();
        rd.close();
        return allLines;

Where 'parameters' ranges from nothing to every URL-encoded parameter from the HTML I posted, and connection is (currently, as in I've tried other configurations) set up like:

        connection = (HttpURLConnection) url.openConnection();
        HttpURLConnection.setFollowRedirects(true);
        connection.setDoOutput(true);
        connection.setDoInput(true);
        connection.setRequestMethod("POST");
        connection.setAllowUserInteraction(false);
        connection.setRequestProperty("Content-type", "application/x-www-form-urlencoded; charset=" + "UTF-8");
        connection.setUseCaches(false);
        connection.connect();

I know the details are a little sparse, so let me know if you need any more info and thanks very much for any input!

Answer 1

You need to ensure that you also take the name=value pairs of all <input type="hidden"> elements and the one <input type="submit"> you'd like to "press" programmatically in the query string. The button has only a name, but regardless, you need to pass it along. This way the server side can distinguish if a button is pressed and if so, which one.

Besides, if the form requires a session (it might use a request based token), then you need to maintain the cookies yourself as well. Check and get the Set-Cookie from the header of the first response and set it as Cookie header of the subsequent requests.

For easier and less verbose HTTP/form processing, consider Apache HttpComponents Client.

See also:

• How to use java.net.URLConnection

Answer 2

Your code may deadlock in certain circumstances. You should close the output before you read from input. Without closing the stream, server doesn't know your request is complete.

If you need to keep output open for some reason, you need to send "Content-length" header to inform server how much to expect.

Parameters must be properly encoded. I normally put in a map and serialize it like this,

            StringBuilder sb = new StringBuilder(baseUrl);

            for (Parameter p : parameters) {
                    if (sb.length() > 0)
                        sb.append('&');
                    try {
                            sb.append(URLEncoder.encode(p.name, "UTF-8"));
                            if (p.value != null) {
                                sb.append('=');
                                sb.append(URLEncoder.encode(p.value, "UTF-8"));
                            }
                    } catch (UnsupportedEncodingException e) {
                        // Not really possible, throw unchecked exception
                        throw new IllegalStateException("No UTF-8");
                    }
            }

The best way to figure out what parameters to post is to capture a trace of the posting in browser and see how browser does it. FireBug/DebugBar are your friends.