It is impossible to make a general statement about the performance of .NET 4.0 compared to earlier versions. Microsoft is constantly improving the framework, thus making performance improvements. Performance is improved both in the BCL as in the runtime itself. For instance, cold startup time is improved dramatically and the performance of the GC has improved both in 3.5sp1 and in 4.0. On the other hand, the framework is getting bigger and this means more code. More code means more page misses, means slower performance.
On the other hand, .NET 4.0 introduces new techniques that that can drastically improve the performance of your code (such as the TPL). And don’t forget that new techniques will be introduced that improve the developer performance, which is the most important part of all.
So don't get influenced by sentiment and gossip. Even if .NET 4.0 is slower, this shouldn't worry you. You should benchmark if that part of the framework is fast enough for your requirements. If it isn't, you can always come back here at SO and ask how that particular scenario can be improved. It is almost always possible to get around it.