C# downloading a webpage. A better way needed, CPU usage high

Question

I'm trying to get this piece of code working a little better. I suspect it's the loop reading one byte at a time. I couldn't find another way of doing this with gzip decompression. Implementing a StreamReader is fine, but it returns a string which I can't pass to the decompression stream.

Is there a better way?

byte[] bufffer = null;
List<byte> resourceBytes = new List<byte>();
int byteValue = 0;
WebResource resource = new WebResource();
HttpWebResponse webResponse = null;

try {
    HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(resourceUri);
    webRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
    webRequest.Headers.Add(HttpRequestHeader.AcceptCharset, "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
    webRequest.UserAgent = agent;
    webRequest.Accept = "text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1";
    webRequest.Credentials = CredentialCache.DefaultCredentials;
    webRequest.Referer = resourceUri.OriginalString;
    webRequest.Timeout = 5000;

    webResponse = (HttpWebResponse)webRequest.GetResponse();

    Stream webStream = webResponse.GetResponseStream();

    if (!string.IsNullOrEmpty(webResponse.ContentEncoding)) {
        if (webResponse.ContentEncoding.ToLower().Contains("gzip")) {
            webStream = new GZipStream(webStream, CompressionMode.Decompress);
        }
        else if (webResponse.ContentEncoding.ToLower().Contains("deflate")) {
            webStream = new DeflateStream(webStream, CompressionMode.Decompress);
        }
    }

    do {
        byteValue = webStream.ReadByte();

        if (byteValue != -1) {
            resourceBytes.Add((byte)byteValue);
        }

    } while (byteValue != -1);


    //Free up resources
    webStream.Close();
    webResponse.Close();

    bufffer = resourceBytes.ToArray();

Answer 1

Is the WebClient class no use for what you want to do?

Answer 2

If you want the response as a string you can do this.

String ReponseText;

IO.StreamReader ResponseReader = New IO.StreamReader(webStream );
ReponseText= ResponseReader.ReadToEnd();

If you want an actual Byte Array do this (Sorry, Don't feel like converting to C# for this one)

'Declare Array Same size as response
Dim ResponseData(webStream .Length) As Byte 
'Read all the data at once
webStream.Read(ResponseData, 0, webStream .Length)

Answer 3

Kibbee, the GZipStream class only accepts a Stream object and Stream objects can only be read one byte at a time.

If you try to read in chunks faster than the data is comming it, it will fall over.

Answer 4

I'd agree with jmcd that WebClient would be far simpler, in particular WebClient.DownloadData.

re the actual question, the problem is that you are reading single bytes, when you should probably have a fixed buffer, and loop - i.e.

int bytesRead;
byte[] buffer = new byte[1024];
while((bytesRead = webStream.Read(buffer, 0, buffer.Length)) > 0) {
  // process "bytesRead" worth of data from "buffer"
}

[edit to add emphasis] The important bit is that you only process "bytesRead" worth of data each time; everything beyond there is garbage.

Answer 5

You're not understanding what I'm saying.

The GZipStream class accepts a stream as an argument. Then you use GZipStream to read the decompressed data. unfortunately, GZipStream only allows you to read one byte at a time (reliably).

Reading blocks of data is only good if your connection can keep up, otherwise the the Length property will be 0 even though more data is on its way.