.Net Web Request - Unable to connect to the remote server

Question

I'm using .Net Web Request to read the html of selected Webpages.

Once in a while I'm getting the following error:

2010-09-05 13:14:51,986 [File_29] ERROR Boxer.Classes.GetLinks [(null)] - HtmlUtils - GetResponse - Url - http://domain.com/index.html
System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 69.212.110.118:5068
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.GetResponse()
   at Boxer.Classes.GetLinks.GetResponse(String url, Int32 timeout, Boolean& timeoutException)

I know for sure that the target URL is available.

Any idea what is the cause of the error?

Answer 1

There could be different reasons for this happening. Maybe the target server is throttling the response because it believes the request comes from a bot. In this case you might consider sending proper HTTP headers (if it works in your browser, try sending the same request headers). Also there could be a proxy in between that you need to configure and use.