Why does WebClient timeout when trying to crawl this shortened URL??

Question

All,

I have a small webcrawler that sometimes has to crawl twitter and pull out URL's. I use a modified version of the Webclient class provided in the .net framework.

Normally this works fine, even with shortened URL's from sites such as bit.ly.

However, with the following url: http://is.gd/CioW The webclient times out.

Its meant to redirect you to here: http://digg.com/microsoft/Less_Virtual_More_Machine_Windows_7_and_the_magic_of_Boot

You think they're filtering certain clients?

Any ideas as to how I can fix this or why its happening?

Answer 1

Are you sure you can hit that URL from your network, without going through a proxy?

Does your webclient control follow redirects? You could test this by creating a TinyURL and see if your webclient can browse to it.

If you are going through a proxy in your browser, you'll need to set it up in the WebClient control.

It should be easy to test if they are filtering clients - set the UserAgent on the Request object to match that of FireFox for example.