Safest way would be to load that plugin into a separate AppDomain which you configure with the security evidence for the requirements you have.
When you create an AppDomain, you can specify exactly the kinds of things code can do in this sandbox. Code that runs there is restricted to the limits you set. But this process can be confusing the first time you do it and may still leave you open to vulnerabilities.
Using AppDomains to isolate assemblies is an interesting process. You'd think you load your plugins into the other AppDomain then use them via proxies in your AppDomain, but its the other way around. They need to use your proxies in their AppDomain. If you fail to understand and do this right, you'll end up loading your plugin code within your main AppDomain and executing it there instead of in the restricted domain. There are lots of gotchas that you'll get bit by (subscribing to events has some interesting side effects) if you don't do things correctly.
I'd suggest prototyping, brush up on the AppDomain chapter in CLR Via C#, and read as much as you can on the subject.
Here's a test app I made to investigate cross-appdomain events.
http://cid-f8be9de57b85cc35.skydrive.live.com/self.aspx/Public/appdomainevents.rar