Firstly, Java and JavaScript are completely different and unrelated languages. Firefox uses JavaScript; it does not use Java at all.
Secondly, this was not done for performance reasons, it was done to make it simpler to write add-ons and extensions that can be used with Firefox on any platform. C++ code needs to be compiled, and once compiled will only run on the platform that it was compiled for. JavaScript is an interpreted language which runs in an interpreter embedded in the browser, and so the same extensions can generally run on MacOS, Linux, or Windows (or anything else that Firefox runs on).
And yes, there are other programs that use scripting languages for this sort of purpose (to allow easy customization and extension). For example, Civilization IV's UI and game logic is written entirely with Python and XML for this reason, while the performance-intensive graphics code is still in C++.