There isn't really a language-agnostic answer to this question, but I'll assume that the plugin is some kind of dll.
If you can avoid trusting the plugin, there are some potential benefits from avoiding it. If nothing else it helps isolate bugs, but also it provides your user with some protection from malicious plugins, and helps with the reliability and robustness of your app.
However, it is fairly difficult to avoid trusting a plugin. Depending on OS, are you going to run it in its own process with minimal privileges, no filesystem or sockets access, restricted access to resources such as memory and CPU time, a monitor that will kill it if it appears unresponsive, communicating with your process only through a pipe? If not, then some buggy plugin will find a way to ruin your afternoon, meaning that you've "relied on" its correctness whether you intended to or not.
Catching exceptions implies that you're running the plugin in one of your application threads. So you are trusting it. You can't "make sure it doesn't crash" if it has the ability to do some OS-specific thing that will make the kernel shut down your process with extreme prejudice.
All that said - on the specific issue of catching exceptions, callers should catch exceptions if and only if they can do something useful with them.
I'd catch them if the plugin is doing something non-essential (e.g. rendering one component of a web page), or let them go if the plugin is doing something absolutely essential to the program (for example if it's a video codec in a command-line transcoding program, maybe I'd catch the exception and rethrow it after logging an error "blaming" the fault on the plugin. Otherwise I'd handle it the same as any other unexpected error in the program).