Many questions have been asked about this, and many answers have been proposed, but nothing's worked for me.
At one point, I turned on .NET symbol loading for WPF projects, using these instructions: http://blogs.msdn.com/sburke/archive/2008/01/16/configuring-visual-studio-to-debug-net-framework-source-code.aspx
Worked great, all cool and that.
Except in Silverlight projects, it would also try to load symbols from that server, but they didn't exist, so startup time to debug a project was huge.
So, I turned off symbol loading, and it did stop trying to load all the symbols, except for one namespace: System.Windows.Browser. VS still tries to load those with each F5, taking several seconds and driving me crazy.
I tried to add the symbol server mentioned here as well: http://blogs.msdn.com/seema/archive/2008/10/08/xperf-a-cpu-sampler-for-silverlight.aspx
Thinking that maybe the symbols were there, they'd get loaded and cached, and all would be well. But no, not only do they not get loaded, but VS tries to load symbols for every other DLL in my project, including those for which symbols have never been available, making it take even longer to start up.
Frankly, I don't even need to debug into the sources of these DLLs, I just thought it would be cool. Now I can't turn it off and it's driving me batty. What's the fix?