You're certainly not going to want to do this in safe code, unless you've got a lot of buffer memory, as the encoder would run glacially slowly.
One option is to run with elevated permissions, and then you can interact with external COM packages - see http://forums.silverlight.net/forums/p/156112/350144.aspx for some useful links
Here's a link from stackoverflow on the general issue that says pretty much the same thing - http://stackoverflow.com/questions/2370880/using-native-dlls-in-c-com-wrapper-and-use-dll-in-silverlight
If you can create a C# wrapper around calls to any native encoding DLL of your choice, then you can put the wrapper assembly in the GAC and you should be good to go - this would of course mean you need an additional install step to get this assembly into the gac, which is outside the 'normal' silverlight experience
To make this separation easier to implement and use, we introduced what we call the simple sandboxing APIs in the .NET Framework 2.0, which create each application domain with a given permission set for its sandbox and a list of fully trusted assemblies that are not in the Global Assembly Cache (GAC), as all assemblies in the GAC are already fully trusted.
taken from http://msdn.microsoft.com/en-us/magazine/cc765416.aspx - dated, but I believe still accurage