As other folks have pointed out, this is the case for all browser plugins, including Java and Flash. Even with the download requirement, RiaStats indicates that SL is now installed on ~60% of all browsers, up from ~25% a year ago, so adoption is definitely increasing.
Obviously it's a bit of a chicken-and-egg problem: people will install it when more developers are using it; more developers will use it when more people have it installed. But MS is addressing both sides of the equation in various ways. On the user-adoption side, they're jump-starting the process by requiring it for more and more content throughout their (very large) network of websites. On the developer side, they're working very hard on their developer and designer tools, to make it a much more enjoyable environment to code in than Flash or Java. (Among oer things, ActionScript is a pain; C# is awesome, and it's not even your only choice. You can use VB, or any of the DLR languages like IronRuby or IronPython, or even that weird but oddly attractive beast, F#.)
It'll be a year or two before it hits 90% of all browsers, but I have little doubt it's gonna get there.