Export information/services into JMX and then use RMI+SSL to access it remotely. Your situation is what JMX is designed for (the M stands for Management).
Kevin
2009-05-28 19:39:59
A:
I agree that common metrics ought to be exposed via JMX. We're actually using a lightweight production profiler (Wily), but it isn't very good at capturing state information, and can only perform well when limited to coarse-grained traces. The other issue is that some of these apps are (partially) third-party, so we'll be debugging with decompiled source at best.
ShabbyDoo
2009-05-28 19:53:46
I still maintain that hooking up your debugger to a production application is a bad idea. Users are going to have no idea whats going on when you hit a breakpoint and spend a few moments grepping through memory.I would try to find the spots in your code that are having problems and expose the current state via JMX and keep detailed audit logs of what is happening.
Kevin
2009-05-28 20:05:01
We'd actually be taking the badly behaving instance out of our load-balanced cluster before debugging. I do agree that debugging an app w/ active users would be a really bad idea.
ShabbyDoo
2009-05-28 20:25:16
+2
A:
You're absolutely right: the Java Debugging API is inherently insecure. You can, however, limit it to UNIX domain sockets, and write a proxy with SSL/SSH to let you have authenticated and encrypted external connections that are then proxied into the UNIX domain socket. That at least reduces your exposure to someone who can get a process into the server, or someone who can crack your SSL.
Charlie Martin
2009-05-28 19:40:09
Can one map a port on the default interface to a domain socket? The problem I have (which I found out after my initial post) is that the 1.4.x Sun JVM can only bind to the default(?) interface. Therefore, some magical mapping would be required so that this port wouldn't be exposed outside the VM.
ShabbyDoo
2009-05-28 23:16:18
A:
Good question.
I'm not aware of any built-in ability to encrypt connections to the debugging port.
There may be a much better/easier solution, but I would do the following:
- Put the production machine behind a firewall that blocks access to the debugging port(s).
- Run a proxy process on the host itself that connects to the port, and encrypts the input and output from the socket.
- Run a proxy client on the debugging workstation that also encrypts/decrypts the input. Have this connect to the server proxy. Communication between them would be encrypted.
- Connect your debugger to the proxy client.
Jared
2009-05-28 19:42:35
As a side note: Our production servers are behind a firewall, but they are exposed to some internal network segments.
ShabbyDoo
2009-05-28 20:03:57
+1
A:
If you use SSH you can allow tunneling and tunnel a port to your local host. No development required, all done using sshd, ssh and/or putty.
The debug socket on your java server can be set up on the local interface 127.0.0.1.
KarlP
2009-05-28 19:52:32
If this works (testing now), it seems like the best option for us. It's not like we debug regularly, but we do want the ability to catch a JVM in a misbehaving state.
ShabbyDoo
2009-05-28 20:27:34
I think this will only work on JDK 1.5+:http://java.sun.com/j2se/1.5.0/docs/guide/jpda/enhancements.htmlSee "the dt_socket transport has been amended to take a local address when running in server mode" in the above link.
ShabbyDoo
2009-05-28 20:39:25
@Shabby - yep - looks like this would work 1.5+, and is a great solution. The alternate is to lock down the debug ports via a firewall (software or hardware.) Maybe checkout ipchains for your linux hosts? (http://tldp.org/HOWTO/IPCHAINS-HOWTO.html)
Jared
2009-05-28 21:12:13
One of our infrastructure folks noted that binding to loopback limits access to those with rights on the machine in question -- perhaps a bit broad for some applications. However, this is probably good enough in many cases.
ShabbyDoo
2009-06-01 00:19:39