You can not unload an assembly from an appdomain. You can destroy appdomains, but once an assembly is loaded into an appdomain, it's there for the life of the appdomain.
See Jason Zander's explanation of Why isn't there an Assembly.Unload method?
If you are using 3.5, you can use the AddIn Framework to make it easier to manage/call into different AppDomains (which you can unload, unloading all the assemblies). If you are using versions before that, you need to create a new appdomain yourself to unload it.