Look into the com.apple.eawt package. Specifically, when you initialize your app, do something like the following:
if (System.getProperty("os.name").startsWith("Mac OS X")) {
// only do this setup if we know this is a Mac
com.apple.eawt.Application macApp = com.apple.eawt.Application.getApplication();
java.awt.PopupMenu menu = new java.awt.PopupMenu();
// create your java.awt.MenuItem objects here
// add to menu via java.awt.Menu#add(java.awt.MenuItem)
macApp.setDockMenu(menu);
}
If you are distributing this as a cross-platform application, Apple provides an Apple Java Extensions jar with stubs for the com.apple.eawt package, so the code will compile with non-Apple JDKs.