If you're looking to make some changes to your application, by refactoring it and splitting out parts, consider all of these.
Custom controls get their own control assembly
Definitely create control assemblies for any custom controls you develop. Not only do you get the benefit of self-contained controls, and optional use in your current and future projects, you can
- take advantage of default control styles
- use the cached assembly feature with them
- share components with other projects
- invest in your core code and controls, instead of investing in cleaning up application logic (if you're using static or style analysis, for instance) - spend your time where it will make the most impact
Consider dynamically loading new assemblies
There are some methods available for dynamically loading additional code into your app domain, it may be possible to abstract out less-often used parts of your app, and use this to load in those components. This is a more complex and involved app, but it can improve start-up performance.
It'll take time to split out the code into other assemblies when you're looking at a large application, and testing it can be a challenge. But you can end up with "sub-pages" and parts of your app being loaded only as needed.
Taking the time to design a system to load in new functionality and parts of your app, and architecting this framework, can take time to do right. This typically is using AssemblyPart to load a new assembly that you can reflect into and instantiate new objects from.
Merged Resource Dictionaries
Resource dictionaries can allow you to store styles, control templates, and other resources outside of pages, and outside of your App.xaml.
Cached Assemblies
Once you move to Silverlight 3, you can use the cached assemblies feature to store individual assemblies outside your .Xap, alongside, on your server - and as a bonus, those assemblies will be cached on the machine for quite some time.
A resource diet
Are you really using all of your graphic assets, XAML, controls, string resources, etc., that are stored inside your XAP file? Audit it from time-to-time and make sure you're getting the most bang for your byte.
A splash screen
If you're simply trying to improve the performance (download time) for your application initially, consider creating a splash screen. The Silverlight Toolkit samples have one - it's a simple Silverlight page that will load and display while your .Xap downloads.
Remote graphics
Instead of including image resources right inside your application/XAP, move your images to your CDN or server, so they can be loaded only as needed. This is often a nice and quick win.
Simplify your app
Make sure you actually need it to be XAML-heavy, graphic-heavy, etc. Maybe it can be simplified!