Getting "underlying connection was closed" on HttpWebRequest

Question

Hello all... I have an application written in VB.NET (NOT asp.net, it is a Windows Console app). I am trying to call a url (an html page) and get back the response into a string. The response is straight JSON, no html tags whatsoever. It opens with { and closes with }.

I create the HttpWebRequest object fine. Then call req.GetResponse(). As soon as I do this I get the error The underlying connection was closed: The connection was closed unexpectedly. I've been googling and checking stackoverflow, and tried everything I've found that applies (a lot of it has to do with WCF service configurations, which don't apply).

Here is my code:

Public Function GetJSON(ByRef d As db.Device) As Boolean
    Try
        d.Url = "http://" & d.IpAddress & ini.doc.<svc>.<url>.Value

        Dim req As HttpWebRequest = HttpWebRequest.Create(d.Url)
        // req.Accept = "*/*"
        // req.Timeout = 30000
        // req.ReadWriteTimeout = 30000
        // req.KeepAlive = False
        // req.UseDefaultCredentials = True
        // req.CachePolicy = HttpWebRequest.DefaultCachePolicy
        // req.Proxy = HttpWebRequest.DefaultWebProxy
        // req.ProtocolVersion = New System.Version(1, 0)

        Dim rsp As HttpWebResponse = req.GetResponse()

        Return True
    Catch ex As Exception
        Log(ex.Message)
        Return False
    Finally
        rsp = Nothing
        req = Nothing
    End Try
End Function

The commented out lines (wrong comment style, but so SO will parse it right) are all the things I have tried so far based on what I've found online. None of them fixed it. I have verified that the URL being built is correct; if I call the exact same URL in my browser it gives back exactly the right expected response.

I've tried wiresharking it... and i see the actual expected, complete data come back in the shark output, and then a few lines, and then a red line which says: http > 51943 [RST] Seq=1607 Win=0 Len=0 which is the last line to show up before .NET is throwing the error.

I also tried turning on System.Net tracing/logging per a post here on SO, and in the output file from that I see similarly all the expected JSON data does come back, but after it comes back it throws these lines in the .NET trace log:

System.Net.Sockets Verbose: 0 : [7040] Exiting Socket#60467532::Receive()   -> 1605#1605
System.Net.Sockets Verbose: 0 : [7040] Socket#60467532::Receive()
System.Net.Sockets Verbose: 0 : [7040] Data from Socket#60467532::Receive
System.Net.Sockets Verbose: 0 : [7040] 00000000 :                                                 : 
System.Net.Sockets Verbose: 0 : [7040] Exiting Socket#60467532::Receive()   -> 0#0
System.Net.Sockets Verbose: 0 : [7040] Socket#60467532::Dispose()
System.Net Error: 0 : [7040] Exception in the HttpWebRequest#27806816:: - The underlying connection was closed: The connection was closed unexpectedly.
System.Net Error: 0 : [7040] Exception in the HttpWebRequest#27806816::GetResponse - The underlying connection was closed: The connection was closed unexpectedly.

Any ideas where to go next to try to figure this out? We are reading this data out of some environmental-monitoring sensor devices, and they gave us this url to use.

Two things that really get me and confuse me on this are that
a) it works perfectly fine when called in a browser
b) both WireShark and .NET tracing show all the data actually IS coming back, and the framework is for some reason excepting AFTER receiving all the data!

The WebException itself is being very little use, as its InnerException is null and its Status just says "ConnectionClosed {8}"

Thanks in advance!!!

UPDATE 08/18 1130: I have also tried now using just System.Net.WebRequest as opposed to HttpWebRequest. This did not make any difference either.

UPDATE 08/18 1222: I just tried switching my code instead of using [Http]Web[Request|Response] to dimming up a WebClient object instead and using its DownloadString() method. This however also throws the same exact error.

UPDATE 08/18 1230: Tried using My.Computer.Network.DownloadFile() - also gets the same connection closed error.

Answer 1

Can you post the entire contents of the trace log on pastebin.com and post a link here?

You might be getting this exception because the server might say in "Content-Length" header that it is sending N bytes of entity, but is actually sending less than N bytes and closing the connection.

Answer:

Thanks for the data. From the tracelog and wireshark trace, it seems that the server is not sending any response headers, and directly sending the data. This is a HTTP protocol violation. That is why the client is throwing the exception.

Answer 2

Here is how i got it working... Thanks to feroze for pointing me in the right direction!
(points awarded)

Public Function GetJSON(ByRef d As db.Device) As Boolean
    Try
        Dim tcp = New TcpClient()
        tcp.Connect(d.IpAddress, 80)

        Dim ns = tcp.GetStream()

        Dim req As Byte() = System.Text.Encoding.ASCII.GetBytes(
            "GET /getdata.htm HTTP/1.1" & vbCrLf & vbCrLf
        )
        ns.Write(req, 0, req.Length)

        Dim rsp(2048) As Byte, rcv As Integer
        Do
            rcv = ns.Read(rsp, 0, rsp.Length)
            d.JSON &= System.Text.Encoding.ASCII.GetString(rsp, 0, rcv)
        Loop Until rcv = 0

        tcp.Close()

        Return True
    Catch ex As Exception
        Log(ex.Message)
        Return False
    End Try
End Function