StreamReader ReadToEnd() after HttpWebRequest EndGetResponse() - most scalable?

Question

I am calling a RESTful web service in the back-end of some ASP.NET pages.

I am using ASP.NET asynchronous pages, so under the hood I am using the methods:

HttpWebRequest BeginGetResponse()

and

HttpWebRequest EndGetResponse()

The response string in my case is always a JSON string. I use the following code to read the entire string:

using (StreamReader sr = new StreamReader(myHttpWebResponse.GetResponseStream()))
{
  myObject.JSONData = sr.ReadToEnd();
}

Is this method OK in terms of scalability? I have seen other code samples that instead retrieve the response data in blocks using Read(). My primary goal is scalability, so this back-end call can be made across many concurrent page hits.

Thanks, Frank

Answer 1

It depends on what you mean by "scalable". If you're talking about being able to handle bigger and bigger files, I'd say it's not terribly scalable. Since you're using a single ReadToEnd, a huge stream would require the entire stream be read into memory and then acted upon. As the application streams grow in number, complexity and size you're going to find that this will begin to hamper the server's performance to handle requests. You may also find that your application pool will begin to recycle itself DURING your request (if you end up taking that much virtual memory).

If the stream is always going to be smallish and you're only concerned with the number of streams created, I don't see why this wouldn't scale as long as your streams were dependent on open files, database connections, etc.