There are several approaches you can take to avoid duplication.
The most fundemental approach would be to use two different projects one targeting the .NET Framework, the other targeting Silverlight. Thus their references point at the appropriate set of assemblies and the Silverlight project defines the #SILVERLIGHT
conditional compilation symbol. However they share the same set of code files, that is one will be using linked files. You would toggle around parts the need to be done differently using conditional compilation.
WCF RIA Services also offers a mechanism where you place code in *.shared.cs files in the server side project that will be automatically duplicated into a "Generated_Code" folder in the client side project.
There is a way to share a single set of code and a single output dll between both Silverlight 4 and .NET 4.0. However you will need to limit the references to a very narrow set of files. See Sharing Silverlight Assemblies with .NET Apps