OK, I've found at least the cause to the problem and a somewhat-unsatisfactory workaround for it!!
I started by adding a new unit test project and adding to it single project references to it from my main solution. I discovered that my data (business entities) project suffers from the problem I am describing, and because all my other assemblies are in some way dependant on my data project, the fact that that one recompiles causes all the other ones to.
Why does that one ALWAYS recompile? It's something to do with Entity Framework. Create an empty project, and add an empty EF model to it - it will ALWAYS recompile, even if you haven't changed it. This then has the knock on effect of causing any dependencies to recompile, presumably because the metadata of the data assembly has changed.
The workaround I've found is rather than adding a project reference, to add a direct assembly reference to the data assembly. The downside is that you can't do e.g. Go To Definition in VS2008 to go to the type directly - but given that we rarely do that with that particular assembly, it's not a massive problem.
Doing this I have cut down the build time just to run a single unit test from around ~20 seconds to ~7 seconds - which makes a massive difference when you're writing dozens of unit tests every day!
If anyone has any ideas as to why EF models do this, I'd be really interested to hear...