In both VS 2005 and 2008, the "Deploy" feature found in the .testrunconfig file determines whether the I WILL WRITE ALL MY RESULTS TO DISK feature is turned on or not.
In 2005 you will need to turn it off manually. In VS 2008 it is turned off by default (IIRC).
You may also want to take a look at this blog post that describes how to get rid of the TestResults folder once in a while.
I can't say I've ever experienced that it recompiles everything just to run. In my experience it recompiles just as you would expect the Build command to do.
Another thing I've noticed is that the way you organize your test results (i.e. the output) while the tests run can actually impact performance - e.g. if you categorize the test results, it runs slower, apparantly because it's using up CPU cycles updating the list.