(Ran out of space in a comment, so I'm writing a full answer instead...)
@Dan: I think the underlying need is quite common: we want refactoring (and every other computer operation) to appear instant*. Whenever here is a noticeable delay, we wish for things to be faster, and we start looking for ways to make that happen.
One of the reasons that the built-in refactorings for C# can be slow is that we deliberately chose correctness over performance. We knew that some of our users would be heavy in to TDD, and they would be able to detect and recover from an error in an automated refactoring, but that many users would be at the mercy of our implementation. We figured that if you knew that we might get a refactoring wrong, you'd grow suspicious and stop using the tool. So, we take the time to do detailed validation of the refactoring. (There are other reasons, and it is possible to do fast, reliable refactoring, but shipping is a feature, too.)
In this case @jdk wants to tell Visual Studio "hey, don't worry about these other projects, I'll accept the risk of missing something important". There isn't really a way to do that, I'm afraid.
You didn't say which version of Visual Studio you are using. I'm pretty sure the team has continued to improve the performance of the tools, so you may want to check out the VS 2010 RC (http://msdn.microsoft.com/en-us/vstudio/dd582936.aspx) and see if things are any better.
Some of the 3rd-party refactoring tools out there are faster. I haven't looked closely, but I'm sure that some take the simple approach for performance, and make mistakes in some cases, while others are more reliable. Choose carefully.
*Whenever someone would ask me "is this fast enough?" I asked back "would a user notice a delay?" If yes, then they would like it faster. Any delay is disappointing, just like a missing feature or a bug. Our job is to decide when it's the right time to make more changes and when to ship.