Technically speaking, Silverlight assemblies are similar as normal CLR assemblies, with the exception that they reference different version of runtime (and different version of system libraries such as mscorlib). Silverlight can run only Silverlight assemblies, so if you're compiling F# code you need to instruct the F# compiler to reference Silverlight (here are Visual Studio templates from Luke Hoban and here is a recent sample Silverlight app in F# by Brian McNamara).
Now, regarding the non-F# assemblies, I'm afraid this may be a problem. In principle you don't need to recompile them - there are tools to change the version (to turn CLR assembly into Silverlight assembly). See for example this article. In practice, Silverlight has many limitations (lots of methods missing, you are not allowed to do some Reflection tricks for security reasons etc.). So, I'm afraid that if you simply convert the assembly to Silverlight, it won't really work, but you may still try that... (but be careful - if a referenced method is missing, you won't find this out until Silverlight tries to call it at runtime).
Finally, there is also a problem with communicating with the application running in Silverlight as Silverlight apps have pretty limited capabilities. However, Silverlight 4 RC should allow you to read/write local files when running in the Out-of-browser mode (which may be good enough).
In summary I think that there are a lot of issues that may make it impossible to use Silverlight for this. I would maybe consider doing some more testing on Mono and sending feedback to them (if you find some case where the performance is clearly poor) - From my experience, they can be quite effective in responding to the feedback from users and I have the feeling that F# may be quite interesting thing for Mono team.