How to enable wire logging for a java HttpURLConnection traffic?

Question

I've used Jakarta commons HttpClient in another project and I would like the same wire logging output but using the "standard" HttpUrlConnection.

I've used Fiddler as a proxy but I would like to log the traffic directly from java.

Capturing what goes by the connection input and output streams is not enough because the HTTP headers are written and consumed by the HttpUrlConnection class, so I will not be able to log the headers.

Answer 1

I don't think you can do that automatically, but you could subclass FilterOutputStream and FilterInputStream with the HttpUrlConnection's output and input streams as parameters. Then as bytes are written/read, log them as well as pass them through to the underlying streams.

Answer 2

I've been able to log all SSL traffic implementing my own SSLSocketFactory on top of the default one.

This worked for me because all of our connections are using HTTPS and we can set the socket factory with the method HttpsURLConnection.setSSLSocketFactory.

A more complete solution that enables monitoring on all sockets can be found at http://www.javaspecialists.eu/archive/Issue169.html Thanks to Software Monkey for pointing in the right direction of using Socket.setSocketImplFactory

Here is my not ready for production code:

public class WireLogSSLSocketFactory extends SSLSocketFactory {

    private SSLSocketFactory delegate;

    public WireLogSSLSocketFactory(SSLSocketFactory sf0) {
        this.delegate = sf0;
    }

    public Socket createSocket(Socket s, String host, int port,
            boolean autoClose) throws IOException {
        return new WireLogSocket((SSLSocket) delegate.createSocket(s, host, port, autoClose));
    }

    /*
    ...
    */

    private static class WireLogSocket extends SSLSocket {

        private SSLSocket delegate;

        public WireLogSocket(SSLSocket s) {
            this.delegate = s;
        }

        public OutputStream getOutputStream() throws IOException {
            return new LoggingOutputStream(delegate.getOutputStream());
        }

        /*
        ...
        */

        private static class LoggingOutputStream extends FilterOutputStream {
            private static final Logger logger = Logger.getLogger(WireLogSocket.LoggingOutputStream.class);
            //I'm using a fixed charset because my app always uses the same. 
            private static final String CHARSET = "ISO-8859-1";
            private StringBuffer sb = new StringBuffer();

            public LoggingOutputStream(OutputStream out) {
                super(out);
            }

            public void write(byte[] b, int off, int len)
                    throws IOException {
                sb.append(new String(b, off, len, CHARSET));
                logger.info("\n" + sb.toString());
                out.write(b, off, len);
            }

            public void write(int b) throws IOException {
                sb.append(b);
                logger.info("\n" + sb.toString());
                out.write(b);
            }

            public void close() throws IOException {
                logger.info("\n" + sb.toString());
                super.close();
            }
        }
    }
}

Answer 3

Solution#1: Use Decorator Pattern

You will have to use Decorator pattern on HttpURLConnection class to extend it's functionality. Then override all HttpURLConnection method(s) and delegate operation to Component pointer as well as capture required information and log it.

Also make sure you override parent class URLConnection.getOutputStream() : OutputStream and URLConnection.html#getInputStream() : InputStream methods to return decorated OutputStream and InputStream objects as well.

.

Solution#2: Use custom, in-memory http proxy

Write a simple http proxy server and have it start in it's separate thread during application startup and initialization. See Example simple proxy server.

Have your application configured to use above HTTP proxy for all your requests. See configuring Java to use Proxies.

Now all your traffic is going through above proxy, just like how it happens in fiddler. Hence you have access to raw http stream "from client to server" as well as "back from server to client". You will have to interpret this raw information and log it as required.

Update: Use HTTP Proxy as Adapter to SSL based Web Server.

  == Client System =============================== 
  |                                              | 
  |    ------------------------------            | 
  |   |                              |           | 
  |   |    Java process              |           | 
  |   |                       ----   |           | 
  |   |        ----------    |    |  |           | 
  |   |       |          |    -O  |  |           | 
  |   |       |  Logging |        |  |           | 
  |   |       |   Proxy <---HTTP--   |    -----  | 
  |   |       |  Adapter |           |   |     | | 
  |   |       |  Thread o------------------>   | | 
  |   |       |        o |           |   |     | | 
  |   |        --------|-            |   | Log | | 
  |   |                |             |    -----  | 
  |    ----------------|-------------            | 
  |                    |                         | 
  =====================|========================== 
                       |                           
                       |                           
                     HTTPS                         
                      SSL                          
                       |                           
  == Server System ====|========================== 
  |                    |                         | 
  |    ----------------|----------------         | 
  |   |                V                |        | 
  |   |                                 |        | 
  |   |   Web Server                    |        | 
  |   |                                 |        | 
  |    ---------------------------------         | 
  |                                              | 
  ================================================ 

Answer 4

What about using AspectJ to insert a Pointcut to add logging advice around a method? I believe AspectJ can weave it's way into private/protected methods.

It appears that sun.net.www.protocol.http.HttpURLConnection.writeRequest may call sun.net.www.http.HttpClient.writeRequest which takes the MessageHeader Object as an input so that would be your target.

In the end this might work but will be awfully fragile and only work on the Sun JVM; and really you could only trust the exact version you are using.

Answer 5

On the off chance, that you're only interested in getting at the content on the wire (headers, body etc), you might want to give wireshark a go.

This has the advantage of not having to change any code, though if enabling logging through code was what you're after, this answer is not applicable.