"Unable to connect to remote server fail" in HttpWebRequest

Question

Hello everyone,

I am using VSTS 2008 + C# + .Net 3.5 to develop a console application and I send request to another server (IIS 7.0 on Windows Server 2008). I find when the # of request threads are big (e.g. 2000 threads), the client will receive error "Unable to connect to remote server fail" when invoking response = (HttpWebResponse)request.GetResponse().My confusion is -- I have set timeout to be a large value, but I got such fail message within a minute. I think even if the connection are really larger than what IIS could serve, client should not get such fail message so soon, it should get such message after timeout period. Any comments? Any ideas what is wrong? Any ideas to make more number of concurrent connection being served by IIS 7.0?

Here is my code,

   class Program
    {
        private static int ClientCount = 2000;
        private static string TargetURL = "http://labtest/abc.wmv";
        private static int Timeout = 3600;

        static void PerformanceWorker()
        {
            Stream dataStream = null;
            HttpWebRequest request = null;
            HttpWebResponse response = null;
            StreamReader reader = null;
            try
            {
                request = (HttpWebRequest)WebRequest.Create(TargetURL);
                request.Timeout = Timeout * 1000;
                request.Proxy = null;
                response = (HttpWebResponse)request.GetResponse();
                dataStream = response.GetResponseStream();
                reader = new StreamReader(dataStream);

                // 1 M at one time
                char[] c = new char[1000 * 10];

                while (reader.Read(c, 0, c.Length) > 0)
                {
                    Console.WriteLine(Thread.CurrentThread.ManagedThreadId);
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message + "\n" + ex.StackTrace);
            }
            finally
            {
                if (null != reader)
                {
                    reader.Close();
                }
                if (null != dataStream)
                {
                    dataStream.Close();
                }
                if (null != response)
                {
                    response.Close();
                }
            }
        }

        static void Main(string[] args)
        {
            Thread[] workers = new Thread[ClientCount];
            for (int i = 0; i < ClientCount; i++)
            {
                workers[i] = new Thread((new ThreadStart(PerformanceWorker)));
            }

            for (int i = 0; i < ClientCount; i++)
            {
                workers[i].Start();
            }

            for (int i = 0; i < ClientCount; i++)
            {
                workers[i].Join();
            }           

            return;
        }
    }

thanks in advance, George

Answer 1

I reckon you've maxed out the web site's application pool queue. The default is 1000 requests, you're flooding the server with 2000 requests more or less all at once. Increasing the timeout isn't going to solve this.

Try increasing the Queue Length for the application pool the site resides in.

You should try and capture the underlying HTTP status, that'll give you a clue as to what is really going on.

Update:

When I run your code and try and download a sizeable file (200MB) I get (503) Server Unavailable.. Increasing the size of the Application Pool's request queue solves this (I set mine to 10000).

Only once did I see Unable to connect to remote server and sadly have been unable to replicate. This error sounds like there's something broken at the TCP/IP layer. Can you post the full exception?

Answer 2

Kev answered you question already, I just want to add that creating so many threads is not really good design solution (just context switching overhead is a big minus) plus it won't scale good.

The quick answer would be: use asynchronous operations to read data instead of creating a bunch of threads. Or at least use thread pool (and lower worker thread count). Remember that more connections to one source will only speed things up till some degree. Try benchmarking it and you will see that probably 3-5 connections will work faster that 2000 you are using now.

You can read more about asynchronous client/server architecture (IOCP - input/output completion ports) and its advantages here. You can start from here:

MSDN - Using an Asynchronous Server Socket

MSDN - Asynchronous Server Socket Example

CodeProject - Multi-threaded .NET TCP Server Examples

All of these examples uses lower level TCP object, but it can be applied to WebRequest/WebResponse as well.

UPDATE

To try thread pool version, you can do something like this:

ManualResetEvent[] events = new ManualResetEvent[ClientCount];
for (uint cnt  = 0; cnt < events.Length; cnt++)
{
  events[cnt] = new ManualResetEvent(false);
  ThreadPool.QueueUserWorkItem(obj => PerformanceWorker());
}

WaitHandle.WaitAll(events);

Not tested, may need some adjustment.

Answer 3

Go to Smart Thread Pool and downlod the code. It is an instance thread pool that constrains the number of threads. The .Net Thread pool can be problematic in applications that connect to web servers and SQL servers.

Change the loop to this

static void Main(string[] args)
        {
            var stp = new SmartThreadPool((int) TimeSpan.FromMinutes(5).TotalMilliseconds,
                                          Environment.ProcessorCount - 1, Environment.ProcessorCount - 1);
            stp.Start();
            for (var i = 0; i < ClientCount; i++)
            {
                stp.QueueWorkItem(PerformanceWorker);
            }
            stp.WaitForIdle();
            stp.Shutdown();

            return;
        }

This constrains the thread pool to use 1 thread per proc. Adjust this up until performance starts to degrade. Too many threads are worse than too few. you many find that this is optimal.

Also add this to you config. The value of 100 is a default I use. There is a way to do this in code but the syntax escapes me now.

<system.net>

    <connectionManagement>

        <add address=“*“ maxconnection=“100“ />

    </connectionManagement>

</system.net>