Using Jakarta HttpClient to mimic a POST network request

Question

I try to mimic a network request send by a closed source application. The following is the network request snapshot captured from the closed source application :

- Http: Request, POST /%5bvUpJYKw4QvGRMBmhATUxRwv4JrU9aDnwNEuangVyy6OuHxi2YiY=%5dImage
    Command: POST
  - URI: /%5bvUpJYKw4QvGRMBmhATUxRwv4JrU9aDnwNEuangVyy6OuHxi2YiY=%5dImage?
     Location: /%5bvUpJYKw4QvGRMBmhATUxRwv4JrU9aDnwNEuangVyy6OuHxi2YiY=%5dImage
    ProtocolVersion: HTTP/1.1
  - ContentType:  application/x-www-form-urlencoded
     MediaType:  application/x-www-form-urlencoded
    Cache-Control:  no-cache
    Pragma:  no-cache
    UserAgent:  Mozilla/4.0 (Windows XP 5.1) Java/1.6.0_06
    Host:  www.xxx.com:20000
    Accept:  text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
    Connection:  keep-alive
    ContentLength:  164
    HeaderEnd: CRLF
  - payload: HttpContentType =  application/x-www-form-urlencoded
     [SORT]: 0,1,0,10,5,0,KL,0
     [FIELD]: 33,38,51,58,68,88,78,98,99,101,56,57,69,70,71,72,89,90,91,92,59,60,61,62,79,80,81,82
     [LIST]: 1155.KL,1295.KL,7191.KL,0097.KL,2267.KL

I use httpclient to mimic the above request.

/**
 * @param args the command line arguments
 */
public static void main(String[] args) {
    try {
        // TODO code application logic here
        HttpClient httpClient = new HttpClient();
        httpClient.getParams().setParameter(HttpMethodParams.USER_AGENT, "Mozilla/4.0 (Windows XP 5.1) Java/1.6.0_06");

        PostMethod post = new PostMethod("http://www.xxx.com:20000/%5bvUpJYKw4QvGRMBmhATUxRwv4JrU9aDnwNEuangVyy6OuHxi2YiY=%5dImage?");

        NameValuePair[] data = {
            new NameValuePair("SORT", "0,1,0,10,5,0,KL,0"),
            new NameValuePair("FIELD", "33,38,51,58,68,88,78,98,99,101,56,57,69,70,71,72,89,90,91,92,59,60,61,62,79,80,81,82"),
            new NameValuePair("LIST", "1155.KL,1295.KL,7191.KL,0097.KL,2267.KL")
        };
        post.setRequestBody(data);
        httpClient.executeMethod(post);
        System.out.println(post.getResponseBodyAsString());       
    }
    catch (Exception exp) {
        exp.printStackTrace();
    }
}

However, there are no respond from server side. I try to capture my own network request snapshot :

- Http: Request, POST /%5bvUpJYKw4QvGRMBmhATUxRwv4JrU9aDnwNEuangVyy6OuHxi2YiY=%5dImage 
    Command: POST
  - URI: /%5bvUpJYKw4QvGRMBmhATUxRwv4JrU9aDnwNEuangVyy6OuHxi2YiY=%5dImage
     Location: /%5bvUpJYKw4QvGRMBmhATUxRwv4JrU9aDnwNEuangVyy6OuHxi2YiY=%5dImage 
    ProtocolVersion: HTTP/1.1
    UserAgent:  Mozilla/4.0 (Windows XP 5.1) Java/1.6.0_06
    Host:  www.xxx.com:20000
    ContentLength:  234
  - ContentType:  application/x-www-form-urlencoded
     MediaType:  application/x-www-form-urlencoded
    HeaderEnd: CRLF
  - payload: HttpContentType =  application/x-www-form-urlencoded
     SORT: 0%2C1%2C0%2C10%2C5%2C0%2CKL%2C0
     FIELD: 33%2C38%2C51%2C58%2C68%2C88%2C78%2C98%2C99%2C101%2C56%2C57%2C69%2C70%2C71%2C72%2C89%2C90%2C91%2C92%2C59%2C60%2C61%2C62%2C79%2C80%2C81%2C82
     LIST: 1155.KL%2C1295.KL%2C7191.KL%2C0097.KL%2C2267.KL

It seems that there are two major different :

(1) URI

(2) payload

Is there anything I miss out in my httpclient code?

After several experiment, I realize is that, httpclient perform URL encoding on my payload. It explicitly turns

1155.KL,1295.KL,7191.KL,0097.KL,2267.KL

to

1155.KL%2C1295.KL%2C7191.KL%2C0097.KL%2C2267.KL

To avoid this, I write a test code. The following test code works!

try {
    Socket socket = new Socket("www.xxx.com", 20000);
    PrintWriter out = new PrintWriter(socket.getOutputStream(), true);
    final String body = "[SORT]=0,1,0,10,5,0,KL,0&[FIELD]=33,38,51,58,68,88,78,98,99,101,56,57,69,70,71,72,89,90,91,92,59,60,61,62,79,80,81,82&[LIST]=1155.KL,1295.KL,7191.KL,0097.KL,2267.KL";
    final int length = body.length();
    final String s = "POST /%5bvUpJYKw4QvGRMBmhATUxRwv4JrU9aDnwNEuangVyy6OuHxi2YiY=%5dImage? HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded\r\nCache-Control: no-cache\r\nPragma: no-cache\r\nUser-Agent: Mozilla/4.0 (Windows XP 5.1) Java/1.6.0_06\r\nHost: www.xxx.com:20000\r\nAccept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\nConnection: keep-alive\r\nContent-Length: "+length+"\r\n\r\n" + body;
    out.println(s);

     BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
     while(true) {
          String ss = in.readLine();
          if (ss == null) break;
            System.out.println(ss);
     }

}
catch (Exception exp) {
}

The following is the captured snap-shot :

  Frame: Number = 16, Captured Frame Length = 597, MediaType = ETHERNET 
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[7C-04-20-00-01-00],SourceAddress:[01-00-01-00-00-00]
+ Ipv4: Src = 10.176.181.55, Dest = 202.75.55.23, Next Protocol = TCP, Packet ID = 956, Total IP Length = 583
+ Tcp: Flags=...AP..., SrcPort=49253, DstPort=20000, PayloadLen=543, Seq=3920474750 - 3920475293, Ack=3537289604, Win=4380 (scale factor 0x2) = 17520
- Http: Request, POST /%5bvUpJYKw4QvGRMBmhATUxRwv4JrU9aDnwNEuangVyy6OuHxi2YiY=%5dImage
    Command: POST
  + URI: /%5bvUpJYKw4QvGRMBmhATUxRwv4JrU9aDnwNEuangVyy6OuHxi2YiY=%5dImage?
    ProtocolVersion: HTTP/1.1
  + ContentType:  application/x-www-form-urlencoded
    Cache-Control:  no-cache
    Pragma:  no-cache
    UserAgent:  Mozilla/4.0 (Windows XP 5.1) Java/1.6.0_06
    Host:  n2ntbfd03.asiaebroker.com:20000
    Accept:  text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
    Connection:  keep-alive
    ContentLength:  164
    HeaderEnd: CRLF
  - payload: HttpContentType =  application/x-www-form-urlencoded
     [SORT]: 0,1,0,10,5,0,KL,0
     [FIELD]: 33,38,51,58,68,88,78,98,99,101,56,57,69,70,71,72,89,90,91,92,59,60,61,62,79,80,81,82
     [LIST]: 1155.KL,1295.KL,7191.KL,0097.KL,2267.KL

Answer 1

How did you create the snapshot? What do the brackets around the parameter names mean? Are there multiple values for the same parameter instead of one with comma separated value or should the parameter name be enclosed in brackets?

Answer 2

You are also missing the headers: Accept Cache-Control Pragma

The Accept header alone might be enough for the server to respond with something meaningful. Try something like this for httpclient 4:

   httpclient.addRequestInterceptor(new HttpRequestInterceptor() {
      public void process(final HttpRequest request, final HttpContext context) throws HttpException, IOException {
        request.setHeader("Pragma","no-cache");
        request.setHeader("Cache-Control","no-cache");
        request.setHeader("Accept","text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2");
      }
    });

Or something like this for httpclient 3.x:

method.setRequestHeader("Pragma","no-cache");
method.setRequestHeader("Cache-Control","no-cache");
method.setRequestHeader("Accept","text/html, image/gif, image/jpeg, *; q=.2, */*;

Answer 3

Judging by the way the parameter names are displayed in your capture SORT vs [SORT]. I suspect the server is expecting you to send the array with multiple form controls, like

SORT=0&SORT=1&SORT=0 ...

They probably do this as a security measure because this can't be done with HTML.

EDIT: You need to pass your parameters like this,

    NameValuePair[] data = {
        new NameValuePair("SORT", "0"),
        new NameValuePair("SORT", "1"),
        ...
        new NameValuePair("FIELD", "33"),
        ...
    };