Hi,
what other update sites do you have enabled in your Eclipse installation?
I had similar problem when upgrading from older Mylyn, and found out that it helps to disable other updates sites. When installing new plugins, or even updating old ones, Eclipse has a tendency to contact all enabled update sites. I don't know what exactly it is trying to do, but it helped me to disable all sites except one I wanted to use.
Btw, Eclipse 3.5M6 has option to use only selected update site -- it is great help when updating/installing plugins. I cannot wait to see Eclipse 3.5 final released, just for this feature.