There is no simple yes/no answer to your question, because it really depends: what do you want to secure, and what do you want to secure it from?
For example, I've used SecurityManager to implement IP filtering and allow only whitelisted IP addresses to connect to my application. If you just want to disallow access to disk files, maybe running application as user with lesser privileges is better solution.
If you don't trust third party plugins at all, remember that once you allow execution of plugin code, that plugin can crash your application if it wants to even if you use SecurityManager. If your application loads plugins, maybe whitelisting plugin and checking the list before loading plugin is better solution.
If you decide to use it, you will take a performance hit (since JVM will do more checks), but how fast it will run really depends on your code/configuration that will do the checks. My IP whitelist was pretty fast since it included only single list lookup; if your checks include invoking remote web service and accessing database you can slow things down a lot, but on the other hand, even that should not matter if you have enough hardware and few concurrent users (in other words, if you can afford it).