Visual studio by default copies all dlls' to each project's bin folder. This also includes system dlls' (with the exception of mscorlib.dll and System.dll), such as System.Xml.Linq, System.ComponentModel.Composition (included since SL4) etc.
Since these files are included in each XAP, XAP sizes grow considerably. In my limited testing, setting "Copy local" to false doesn't seem to break anything. Why does Visual Studio add these files to the bin path? Since a user would already have Silverlight, can I assume that these assemblies are already installed in the GAC of each user or am I missing something?