For what it's worth, JavaScript's performance has greatly improved since the emphasis on scripting speed in Browser Wars 2.0; Java has evolved less radically. Directly comparing such fundamentally different languages isn't really practical, though. And to be honest I don't think performance of the core language is usually going to be your main problem if you're doing something as ambitious as a 3D game in the browser.
To add another option, today you would typically do this Flash using one of the 3D libraries.
Java is what you'd use yesterday (its desktop/applet acceptance is on the wane IMO); WebGL is, hopefully, what we'll be using tomorrow. All three are currently very annoying to write 3D apps with. Get a bunch of practice writing some 2D games before going anywhere near 3D in the browser!
Bonus option for if you don't mind arbitrary closed plugins: Unity
Bonus option for if you're insane: JavaScript, doing the rendering itself, to a <canvas>
.