The 2 trick I know how to do that are to replace the standard runtime library or a similar trick to intercept the OS networking calls or to setup a set of finely tuned firewall rules in the kernel.
Both of these techniques are very OS specific and have no way to da that using java.
Your best bet would be to run an existing socks5 wrapper and let that program start the external application like socksify.
I have had mixed experience with this approach, some applications work, others do not, and never found any logic in it.
YMMV
Another approach is to play a tcpproxy in the Java application (e.g. using the Apache MINA stuff) and have the application connect to your proxy port on localhost. Again this will only work for certain services.