I am comfortable with both Java and C#. Both with Eclipse and VisualStudio. So except I'll have to annoy people with installing Silverlight, is there something that should make me use GWT instead?
GWT is JavaScript, and JavaScript is a standard. If someone wanted to drastically change JavaScript, they'd have a hell of a time doing it. Furthermore, GWT is open-source, so you can tell what they're doing and how they're doing it, if you have the time and inclination to investigate.
Silverlight is closed-source and proprietary, and as such is completely at the whim of one company, Microsoft. If you trust Microsoft not to drastically change the inner workings of Silverlight in a way that negatively effects your development, then this might not matter. But I personally don't.
Also, as you said, Silverlight requires a plug-in to use, which will be annoying to users not only the first time they have to download the plug-in, but also any time they have to update it to use new features, forever.