Chris Thompson mentions browser caching, but that's easily fixable in the browser. What isn't fixable on switching everything to HTTPS is proxy caching. Because HTTPS is encrypted end-to-end, transparent HTTP proxies don't work. There are a lot of places where transparent proxying can speed things up (for instance at NAT boundaries).
Dealing with the additional bandwidth from losing transparent proxying is probably doable - allegedly HTTP traffic is trivial compared with p2p anyway, so it's not as if transparent proxies are the only thing keeping the internet online. It will hit latency irrevocably, and make a slashdotting even worse than it is currently. But then with cloud hosting, both those might be dealt with by tech. Of course "secure server" takes on a different meaning with cloud hosting, or even with other forms of de-centralisation of content across the network like akamai.
I don't think the CPU overhead is that significant. Sure, if your server is currently CPU bound at least some of the time, then switching all traffic from HTTP to HTTPS will kill it stone dead. Some servers may decide that HTTPS is not worth the monetary cost of a CPU that can handle the load, and they will prevent literally everyone adopting it. But I doubt it will be a major barrier for long. For instance, Google has crossed it already and happily serves apps (although not searches) as https without fuss. And the more work servers are doing per connection, the less proportional extra work is required to SSL-secure that connection. SSL can be and is hardware accelerated where necessary.
There's also the management/economic problem that HTTPS relies on trusted CAs, and trusted CAs cost money. There are other ways to design a PKI than the one SSL actually uses, but there are reasons SSL works how it does. For example SSH places the responsibility on the user to obtain a key fingerprint from the server by a secure side-channel, and this is the result: some users don't think that level of inconvenience is justified by its security purpose. If users don't want security, then they won't get it unless it's impossible for them to avoid it.
If users just auto-click "accept" for untrusted SSL certificates, then you pretty much might as well not have it, since these days a man-in-the-middle attack is not significantly more difficult than plain eavesdropping. So, again, there's a significant block of servers which just aren't interesting in paying for (working) HTTPS.