views:

1331

answers:

2

I am hosting a WCF service in IIS using basicHttpBinding. The WCF web services queries back end SQL Server 2008 by using ADO.Net and return a DataTable to client side of WCF service.

I find when the returned DataTable is big, there will be exception said http connection is closed by IIS. Any ideas what is wrong and how to set bigger response size? Another idea is whether there is any hard limitation for the serialization size of an object (I thought maybe the DataTable instance is too big to serialize?)

The error information returned by IIS is:

Exception message:

An error occurred while receiving the HTTP response to http://labmachine1/service.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details.

{"The underlying connection was closed: An unexpected error occurred on a receive."}

Here is my whole source code for server side, for the server I host in web.config, no change for default values. Since I host in IIS, I am using basicHttpBinding.

public class StudentManagement : IStudentManagement
{
    public DataTable Poll(int Id)
    {
        return MakeParentTable();
    }

    private DataTable MakeParentTable()
    {
        // Create a new DataTable.
        System.Data.DataTable table = new DataTable("ParentTable");
        // Declare variables for DataColumn and DataRow objects.
        DataColumn column;
        DataRow row;

        // Create new DataColumn, set DataType, 
        // ColumnName and add to DataTable.    
        column = new DataColumn();
        column.DataType = System.Type.GetType("System.Int32");
        column.ColumnName = "id";
        column.ReadOnly = true;
        column.Unique = true;
        // Add the Column to the DataColumnCollection.
        table.Columns.Add(column);

        // Create second column.
        column = new DataColumn();
        column.DataType = System.Type.GetType("System.String");
        column.ColumnName = "ParentItem";
        column.AutoIncrement = false;
        column.Caption = "ParentItem";
        column.ReadOnly = false;
        column.Unique = false;
        // Add the column to the table.
        table.Columns.Add(column);

        // Make the ID column the primary key column.
        DataColumn[] PrimaryKeyColumns = new DataColumn[1];
        PrimaryKeyColumns[0] = table.Columns["id"];
        table.PrimaryKey = PrimaryKeyColumns;

        // Create three new DataRow objects and add 
        // them to the DataTable
        for (int i = 0; i <= 1000000; i++)
        {
            row = table.NewRow();
            row["id"] = i;
            row["ParentItem"] = "ParentItem " + i;
            table.Rows.Add(row);
        }

        return table;
    }
}

Client side code:

static void Main(string[] args)
{
    StudentIdentifier identifier = new StudentIdentifier();
    identifier.Id = 100;
    StudentManagementClient client = new StudentManagementClient();

    DataTable student = client.Poll(identifier);

    Console.WriteLine(student.Rows.Count);
}

Client side web.config:

<binding name="BasicHttpBinding_IStudentManagement" closeTimeout="00:01:00"
    openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
    allowCookies="false" bypassProxyOnLocal="false" 
    hostNameComparisonMode="StrongWildcard"
    maxBufferSize="1000000000" maxBufferPoolSize="1000000000" 
    maxReceivedMessageSize="1000000000"
    messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"
    useDefaultWebProxy="true">
    <readerQuotas maxDepth="32" maxStringContentLength="1000000000" 
        maxArrayLength="1000000000"
        maxBytesPerRead="4096" maxNameTableCharCount="16384" />
</binding>

EDIT 2:

Streaming mode configuration for client side app.config,

    <basicHttpBinding>
      <binding name="BasicHttpBinding_IStudentManagement" closeTimeout="00:01:00"
        openTimeout="00:20:00" receiveTimeout="01:00:00" sendTimeout="01:00:00"
        allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
        maxBufferSize="1500000000" maxBufferPoolSize="1500000000" maxReceivedMessageSize="1500000000"
        messageEncoding="Mtom" textEncoding="utf-8" transferMode="Streamed"
        useDefaultWebProxy="true">
        <readerQuotas maxDepth="1500000000" maxStringContentLength="1500000000"
          maxArrayLength="1500000000" maxBytesPerRead="1500000000" maxNameTableCharCount="1500000000" />
        <security mode="None">
          <transport clientCredentialType="None" proxyCredentialType="None"
            realm="" />
          <message clientCredentialType="UserName" algorithmSuite="Default" />
        </security>
      </binding>
    </basicHttpBinding>

Server side web.config for streaming mode,

  <basicHttpBinding>
    <binding name="BasicHttpBinding_IStudentManagement" closeTimeout="00:01:00"
        openTimeout="00:20:00" receiveTimeout="01:00:00" sendTimeout="01:00:00"
        allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
        maxBufferSize="1000000000" maxBufferPoolSize="1000000000" maxReceivedMessageSize="1000000000"
        messageEncoding="Mtom" textEncoding="utf-8" transferMode="Streamed"
        useDefaultWebProxy="true">
      <readerQuotas maxDepth="32" maxStringContentLength="1000000000" maxArrayLength="1000000000"
          maxBytesPerRead="4096" maxNameTableCharCount="16384" />
      <security mode="None">
        <transport clientCredentialType="None" proxyCredentialType="None"
            realm="" />
        <message clientCredentialType="UserName" algorithmSuite="Default" />
      </security>
    </binding>
  </basicHttpBinding>
+3  A: 

There's a multitude of buffer sizes you can play with - they are kept fairly small (64K) by default in order to avoid Denial-of-service attacks, but if you need to, you can increase those:

 <system.serviceModel>
    <bindings>
        <basicHttpBinding>
            <binding name="largebuffers" closeTimeout="00:01:00"
                openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
                bypassProxyOnLocal="false" transactionFlow="false" 
                hostNameComparisonMode="StrongWildcard"
                maxBufferSize="524288" maxBufferPoolSize="524288" 
                maxReceivedMessageSize="65536"
                messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true"
                allowCookies="false">
                <readerQuotas maxDepth="32" maxStringContentLength="8192" 
                              maxArrayLength="16384" maxBytesPerRead="4096" 
                              maxNameTableCharCount="16384" />
            </binding>
        </basicHttpBinding>
    </bindings>

Have a look at the "MaxBufferSize", "MaxBufferPoolSize", "MaxReceivedMessageSize" settingsin the binding, as well as the various settings in the <readerQuotas> section.

I would try to just increase the "MaxBufferSize" and "MaxBufferPoolSize" first and see if that alone helps - most of the other settings should really be more geared towards when the service receives and needs to process a message.

There are no hard limits as to how big an object can be that you serialize back. However, the DataTable does carry a significant overhead, and if you don't really need that DataTable functionality on the client side, you might also do a conversion on the server to send back just a collection or generic list of objects - instead of the heavy-lift DataTable object.

Furthermore, if you frequently send back large amounts of data, you might need to investigate the streaming capabilities of WCF, too (e.g. if you return pictures, videos, that kind of stuff). Marc

marc_s
Thanks Marc, 1. in your sample configuration, I did not find MaxBufferSize. 2. I think MaxReceivedMessageSize only affects request size and not response size?
George2
Hi George: added the MaxBufferSize - and yes, MaxReceivedMEssageSize probably only affects the messages received by the serviec - that's why I said to increase the MaxBufferSize (and MaxBufferPoolSize) first and see if that solves the problem already.
marc_s
Thanks Marc, I should set transfermode to Buffered? Streamed? Or some other value in order to enable the setting of "MaxBufferSize" and "MaxBufferPoolSize"?
George2
I have tried to increast increase the "MaxBufferSize" and "MaxBufferPoolSize", but does not work, still the same issue. Any ideas what is wrong?
George2
Well, then increase the other values, too! maxReceivedMessageSize, MaxStringContentLength, MaxArrayLength - and try again
marc_s
Sorry Marc, still not working or maybe I did not catch you correctly. Anyway, code talks better, I have posted my code, both client and server side. Detailed exception message. I have not changed web.config at server side, and I only change app.config at client side (following your advice) and I have posted client side app.config. Any ideas what is wrong?
George2
Hi Marc, I have some new findings and want to share here. I increase receiveTimeout/sendTimeout at both client and server side to customize basicHttpBinding behavior, then it works! But when I increase the # of rows returned to be 10,000,000, even if I set timeout value to be 1 hour, the same error happens. Any ideas? Could you reproduce the same issue at your side?
George2
That of course is the other option - it's not the SIZE that matters, but in the end, the time it takes to get the DataTable set up and sent back. In that case, you'll need to adjust the "sendTimeout" and possibly "receiveTimeout" - and that should be fine then.
marc_s
WCF gives you a ton of flexibility in setting up your operations - and possibly you'll have to tweak a lot of settings to get everything right - but at least you **CAN** tweak! Other systems don't give you that option...
marc_s
Thanks Marc, I have tested 1M rows works. But 10M rows never work even if in streaming mode. I think using streaming mode should enable us to transport any size of data. But actually it is not working. I have posted my client side app.config and server side web.config into the EDIT2 section of my original post. Any ideas what is wrong?
George2
No, sorry, just from the information I have, I cannot determine what's wrong. What error (if any) do you get? since you probably only want to stream the response (right?), you could try transferMode="StreamedResponse"
marc_s
Hi Marc, from EDIT 2 section of my original reply, you can see I already enabled Streamed mode, but seems streamed mode does not take any effect, memory still goes up... Any ideas what is wrong?
George2
+1  A: 

George, be sure to look in the event log when things like this happen. See if WCF or any other component has logged an error.

Also, try turning on WCF tracing and see whether there's an internal error that's not being logged somehow. You may need to make the trace verbose in order to see it, but take a look and see what's there.

Just because you get an obscure error from IIS, doesn't mean there's not more and better information available somewhere. After all, the error message did say, "See server logs for more details."

John Saunders
Hi John, I have some new findings and want to share here. I increase receiveTimeout/sendTimeout at both client and server side to customize basicHttpBinding behavior, then it works!But when I increase the # of rows returned to be 10,000,000, even if I set timeout value to be 1 hour, the same error happens. Any ideas? Could you reproduce the same issue at your side?
George2
I have searched in event log, the same error exception information as I posted here. So, no more information. :-(For WCF tracing, you mean?
George2
George, are the client and the service both running in IIS on the same machine? The message you posted speaks about an error _receiving_ a response. I want you to find what's in the event log from the _service_ side. I want to know if the _service_ had any problems.
John Saunders
Also, if you're returning 10,000,000 rows, then you need to use streaming. I bet you're running out of memory in the service, trying to get all 10,000,000 rows-worth of XML into memory at one time.
John Saunders
Thanks for your advice, John. 1. I want to confirm with you that Buffer mode means all data needs to be available in memory at server side before sending to client, and all data needs to be recieved in buffer at client side before deserialization? 2. I am confused why there is no detailed message dump out?
George2
John, I have made further testing that you are correct that memory is almost used up before exception is thrown. I want to have a try about streaming mode, but I have not used streaming mode before, appreciate if you could recommend me some simple tutorial about streaming mode, is it hard? Need to change my server or client side business logics code? :-)
George2
@George2: see the many comments in http://stackoverflow.com/questions/1281934/how-to-add-smart-tags-to-my-net-component/1283538#1283538, and you'll realize why I'd like you to spend two minutes to go to msdn.microsoft.com, and to type "wcf streaming" into the search box and then click "Search". If you don't find what you're looking for in two minutes, come back and say so.
John Saunders
Thanks John, sorry a bit lazy just now. :-)I have implemented streaming quickly and under testing, a quick question, at server side if I return Stream object, who is responsible to Dispose/Close the server side Stream object? I suspect if I close it explicitly, the stream object may be used by underlying channel and impact underlying transfer. Any comments?
George2
George, ask yourself: does it really send a .NET `Stream` object across the wire? No. When you return `Stream`, you're passing it to WCF, which reads the stream and sends the results over the wire. WCF is therefore responsible for disposing on the server. I'm not sure how it works on the client.
John Saunders
Hi John, I have enabled streaming mode, but still the same issue. I think using streaming mode should enable us to transport any size of data. But actually it is not working. I have posted my client side app.config and server side web.config into the EDIT2 section of my original post. Any ideas what is wrong?
George2
"See server logs for more details" -- which Server log does it mean?
George2
George, always check the Application event log on the server. If you're running IIS7, there are some totally new features like Failed Request Tracking, but I haven't used them enough to know whether they would apply.
John Saunders