This might be out of the scope of a Java applet. In fact, global keyboard hooks are definitely out of the scope of simply using Java, but I can help you get moving in the right direction.
However, you have some hope. I'll point you into the direction of JNI (Java Native Interface), which will allow you to use native libraries. Now, since you want to stay in the Java world, I suggest not directly using JNI because you'll have to write some confusing native code (typically C, C++). There are several wrappers for JNI that allow you to use the functions but the native implementations are abstracted away, but a lot of these cost money.
So the best solution for you, I think, is JNA (Java Native Access). This allows you to directly call native libraries from within Java. (NOTE: The implementation not will be cross platform. You have to make separate implementations for Windows, Linux, etc.) There's a good example for Windows keyboard hooks in the examples on the project website.
As for opening it's own window not in a webpage, do you want the applet to not run within the browser but in its own separate process, or just be in a separate window and still rely on the browser window being open?
If you want to simply launch a new window and still require the browser to be open, then here's a good example:
final Frame window = new Frame("This is the Frame's Title Bar!");
window.add(new Label("This is the Frame."));
window.setSize(300,200);
window.setVisible(true);
window.addWindowListener(new WindowAdapter(){
public void windowClosing(WindowEvent we){
window.dispose();
}
});
If you want the applet to spawn a new process and run without the need of the browser, look into JavaFX.