Wcf System.InsufficientMemoryException. Failed to allocate a managed memory buffer of 536870912 bytes.

Question

Hi guys

I have a WCF service method that returns a big byte[] array around 2 mb in size. However when i run that method from the WcfTestClient, I get the following exception.

System.InsufficientMemoryException
Failed to allocate a managed memory buffer of 536870912 bytes. The amount of available memory may be low.

I have searched on google, and people seem to have this issue but I am struggling to find any solution and the actual cause of why is it trying to create such a huge buffer in the first place?

Below are my config settings. I have set the maxItemsInObjectGraph="2147483647" otherwise it was throwing exception about it. The rest is simple i guess.

Server

  <system.serviceModel>
    <services>
      <service behaviorConfiguration="Wcf.ServiceBehavior"
        name="Wcf.WcfService">
        <endpoint address="" binding="wsHttpBinding" contract="Wcf.IWcfService">
          <identity>
            <dns value="localhost" />
          </identity>
        </endpoint>
        <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
      </service>
    </services>
    <behaviors>
      <serviceBehaviors>
        <behavior name="Wcf.ServiceBehavior">
          <serviceMetadata httpGetEnabled="true" />
          <serviceDebug includeExceptionDetailInFaults="true" />
          <dataContractSerializer maxItemsInObjectGraph="2147483647" />
        </behavior>
      </serviceBehaviors>
    </behaviors>

    <diagnostics>    
      <messageLogging maxMessagesToLog="30000"    
              logEntireMessage="true"    
              logMessagesAtServiceLevel="true"    
              logMalformedMessages="true"    
              logMessagesAtTransportLevel="true">    
      </messageLogging>    
    </diagnostics>
  </system.serviceModel>

WcfTestClient Config This is the default config file that is loaded with WcfTestClient

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <system.serviceModel>
        <bindings>
            <wsHttpBinding>
                <binding name="WSHttpBinding_IWcfService" closeTimeout="00:01:00"
                    openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
                    bypassProxyOnLocal="false" transactionFlow="false" hostNameComparisonMode="StrongWildcard"
                    maxBufferPoolSize="524288" maxReceivedMessageSize="65536"
                    messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true"
                    allowCookies="false">
                    <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
                        maxBytesPerRead="4096" maxNameTableCharCount="16384" />
                    <reliableSession ordered="true" inactivityTimeout="00:10:00"
                        enabled="false" />
                    <security mode="Message">
                        <transport clientCredentialType="Windows" proxyCredentialType="None"
                            realm="" />
                        <message clientCredentialType="Windows" negotiateServiceCredential="true"
                            algorithmSuite="Default" />
                    </security>
                </binding>
            </wsHttpBinding>
        </bindings>
        <client>
            <endpoint address="http://localhost:4151/Service1.svc" binding="wsHttpBinding"
                bindingConfiguration="WSHttpBinding_IWcfService" contract="IWcfService"
                name="WSHttpBinding_IWcfService">
                <identity>
                    <dns value="localhost" />
                </identity>
            </endpoint>
        </client>
    </system.serviceModel>
</configuration>

Here is the exception Stack Trace

    <E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent"&gt;

  <System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system"&gt;

    <EventID>131075</EventID>
    <Type>3</Type>
    <SubType Name="Error">0</SubType>
    <Level>2</Level>
    <TimeCreated SystemTime="2010-06-29T09:18:23.9616589Z" />
    <Source Name="System.ServiceModel" />
    <Correlation ActivityID="{7870ff09-e268-4e9d-a692-389fd03db1aa}" />
    <Execution ProcessName="WebDev.WebServer20" ProcessID="4812"
    ThreadID="17" />
    <Channel />
    <Computer>PC-008915</Computer>
  </System>
  <ApplicationData>
    <TraceData>
      <DataItem>
        <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord"
        Severity="Error">
          <TraceIdentifier>
          http://msdn.microsoft.com/en-GB/library/System.ServiceModel.Diagnostics.ThrowingException.aspx&lt;/TraceIdentifier&gt;
          <Description>Throwing an exception.</Description>
          <AppDomain>5ab4f443-1-129222741421051645</AppDomain>
          <Exception>
            <ExceptionType>System.InsufficientMemoryException,
            mscorlib, Version=2.0.0.0, Culture=neutral,
            PublicKeyToken=b77a5c561934e089</ExceptionType>
            <Message>Failed to allocate a managed memory buffer of
            536870912 bytes. The amount of available memory may be
            low.</Message>
            <StackTrace>at
            System.ServiceModel.Diagnostics.Utility.AllocateByteArray(Int32
            size) at
            System.ServiceModel.Channels.BufferManager.GCBufferManager.TakeBuffer(Int32
            bufferSize) at
            System.ServiceModel.Channels.BufferedOutputStream.AllocNextChunk(Int32
            minimumChunkSize) at
            System.ServiceModel.Channels.BufferedOutputStream.WriteCore(Byte[]
            buffer, Int32 offset, Int32 size) at
            System.ServiceModel.Channels.BufferedOutputStream.Write(Byte[]
            buffer, Int32 offset, Int32 size) at
            System.Xml.XmlStreamNodeWriter.FlushBuffer() at
            System.Xml.XmlStreamNodeWriter.GetBuffer(Int32 count,
            Int32&amp; offset) at
            System.Xml.XmlStreamNodeWriter.UnsafeWriteUTF8Chars(Char*
            chars, Int32 charCount) at
            System.Xml.XmlStreamNodeWriter.WriteUTF8Chars(String
            value) at
            System.Xml.XmlUTF8NodeWriter.WriteEndElement(String
            prefix, String localName) at
            System.Xml.XmlSigningNodeWriter.WriteEndElement(String
            prefix, String localName) at
            System.Xml.XmlBaseWriter.WriteEndElement() at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteString(XmlWriterDelegator
            xmlWriter, String value, XmlDictionaryString name,
            XmlDictionaryString ns) at
            WritePropertyDtoToXml(XmlWriterDelegator , Object ,
            XmlObjectSerializerWriteContext , ClassDataContract )
            at
            System.Runtime.Serialization.ClassDataContract.WriteXmlValue(XmlWriterDelegator
            xmlWriter, Object obj, XmlObjectSerializerWriteContext
            context) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerialize(XmlWriterDelegator
            xmlWriter, Object obj, Boolean isDeclaredType, Boolean
            writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle
            declaredTypeHandle) at
            WriteArrayOfPropertyDtoToXml(XmlWriterDelegator ,
            Object , XmlObjectSerializerWriteContext ,
            CollectionDataContract ) at
            System.Runtime.Serialization.CollectionDataContract.WriteXmlValue(XmlWriterDelegator
            xmlWriter, Object obj, XmlObjectSerializerWriteContext
            context) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerialize(XmlWriterDelegator
            xmlWriter, Object obj, Boolean isDeclaredType, Boolean
            writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle
            declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerializeReference(XmlWriterDelegator
            xmlWriter, Object obj, Boolean isDeclaredType, Boolean
            writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle
            declaredTypeHandle) at
            WriteEntityTypeDtoToXml(XmlWriterDelegator , Object ,
            XmlObjectSerializerWriteContext , ClassDataContract )
            at
            System.Runtime.Serialization.ClassDataContract.WriteXmlValue(XmlWriterDelegator
            xmlWriter, Object obj, XmlObjectSerializerWriteContext
            context) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerialize(XmlWriterDelegator
            xmlWriter, Object obj, Boolean isDeclaredType, Boolean
            writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle
            declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerializeReference(XmlWriterDelegator
            xmlWriter, Object obj, Boolean isDeclaredType, Boolean
            writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle
            declaredTypeHandle) at
            WriteNodeTypeDtoToXml(XmlWriterDelegator , Object ,
            XmlObjectSerializerWriteContext , ClassDataContract )
            at
            System.Runtime.Serialization.ClassDataContract.WriteXmlValue(XmlWriterDelegator
            xmlWriter, Object obj, XmlObjectSerializerWriteContext
            context) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerialize(XmlWriterDelegator
            xmlWriter, Object obj, Boolean isDeclaredType, Boolean
            writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle
            declaredTypeHandle) at
            WriteArrayOfNodeTypeDtoToXml(XmlWriterDelegator ,
            Object , XmlObjectSerializerWriteContext ,
            CollectionDataContract ) at
            System.Runtime.Serialization.CollectionDataContract.WriteXmlValue(XmlWriterDelegator
            xmlWriter, Object obj, XmlObjectSerializerWriteContext
            context) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.DataContractSerializer.InternalWriteObjectContent(XmlWriterDelegator
            writer, Object graph) at
            System.Runtime.Serialization.DataContractSerializer.InternalWriteObject(XmlWriterDelegator
            writer, Object graph) at
            System.Runtime.Serialization.XmlObjectSerializer.WriteObjectHandleExceptions(XmlWriterDelegator
            writer, Object graph) at
            System.Runtime.Serialization.XmlObjectSerializer.WriteObject(XmlDictionaryWriter
            writer, Object graph) at
            System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.SerializeParameterPart(XmlDictionaryWriter
            writer, PartInfo part, Object graph) at
            System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.SerializeParameter(XmlDictionaryWriter
            writer, PartInfo part, Object graph) at
            System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.SerializeBody(XmlDictionaryWriter
            writer, MessageVersion version, String action,
            MessageDescription messageDescription, Object
            returnValue, Object[] parameters, Boolean isRequest) at
            System.ServiceModel.Dispatcher.OperationFormatter.SerializeBodyContents(XmlDictionaryWriter
            writer, MessageVersion version, Object[] parameters,
            Object returnValue, Boolean isRequest) at
            System.ServiceModel.Dispatcher.OperationFormatter.OperationFormatterMessage.OperationFormatterBodyWriter.OnWriteBodyContents(XmlDictionaryWriter
            writer) at
            System.ServiceModel.Channels.BodyWriter.WriteBodyContents(XmlDictionaryWriter
            writer) at
            System.ServiceModel.Channels.BodyWriterMessage.OnWriteBodyContents(XmlDictionaryWriter
            writer) at
            System.ServiceModel.Channels.Message.WriteBodyContents(XmlDictionaryWriter
            writer) at
            System.ServiceModel.Security.SecurityAppliedMessage.WriteBodyToSignThenEncryptWithFragments(Stream
            stream, Boolean includeComments, String[]
            inclusivePrefixes, EncryptedData encryptedData,
            SymmetricAlgorithm algorithm, XmlDictionaryWriter
            writer) at
            System.ServiceModel.Security.WSSecurityOneDotZeroSendSecurityHeader.ApplyBodySecurity(XmlDictionaryWriter
            writer, IPrefixGenerator prefixGenerator) at
            System.ServiceModel.Security.SecurityAppliedMessage.OnWriteMessage(XmlDictionaryWriter
            writer) at
            System.ServiceModel.Channels.Message.WriteMessage(XmlDictionaryWriter
            writer) at
            System.ServiceModel.Channels.BufferedMessageWriter.WriteMessage(Message
            message, BufferManager bufferManager, Int32
            initialOffset, Int32 maxSizeQuota) at
            System.ServiceModel.Channels.TextMessageEncoderFactory.TextMessageEncoder.WriteMessage(Message
            message, Int32 maxMessageSize, BufferManager
            bufferManager, Int32 messageOffset) at
            System.ServiceModel.Channels.HttpOutput.SerializeBufferedMessage(Message
            message) at
            System.ServiceModel.Channels.HttpOutput.Send(TimeSpan
            timeout) at
            System.ServiceModel.Channels.HttpRequestContext.OnReply(Message
            message, TimeSpan timeout) at
            System.ServiceModel.Activation.HostedHttpContext.OnReply(Message
            message, TimeSpan timeout) at
            System.ServiceModel.Channels.RequestContextBase.Reply(Message
            message, TimeSpan timeout) at
            System.ServiceModel.Security.SecuritySessionServerSettings.SecuritySessionRequestContext.OnReply(Message
            message, TimeSpan timeout) at
            System.ServiceModel.Channels.RequestContextBase.Reply(Message
            message, TimeSpan timeout) at
            System.ServiceModel.Channels.RequestContextBase.Reply(Message
            message) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.Reply(MessageRpc&amp;
            rpc) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessageCleanup(MessageRpc&amp;
            rpc) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc&amp;
            rpc) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc&amp;
            rpc) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc&amp;
            rpc) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc&amp;
            rpc) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc&amp;
            rpc) at
            System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean
            isOperationContextSet) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.Dispatch(MessageRpc&amp;
            rpc, Boolean isOperationContextSet) at
            System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext
            request, Boolean cleanThread, OperationContext
            currentOperationContext) at
            System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext
            request, OperationContext currentOperationContext) at
            System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult
            result) at
            System.ServiceModel.Dispatcher.ChannelHandler.OnContinueAsyncReceive(Object
            state) at
            System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()
            at
            System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.OnSecurityContextCallback(Object
            o) at
            System.Security.SecurityContext.Run(SecurityContext
            securityContext, ContextCallback callback, Object
            state) at
            System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()
            at
            System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()
            at
            System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object
            state) at
            System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32
            errorCode, UInt32 numBytes, NativeOverlapped*
            nativeOverlapped) at
            System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32
            error, UInt32 bytesRead, NativeOverlapped*
            nativeOverlapped) at
            System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32
            errorCode, UInt32 numBytes, NativeOverlapped*
            pOVERLAP)</StackTrace>
            <ExceptionString>System.InsufficientMemoryException:
            Failed to allocate a managed memory buffer of 536870912
            bytes. The amount of available memory may be low.
            ---&gt; System.OutOfMemoryException: Exception of type
            'System.OutOfMemoryException' was thrown. at
            System.ServiceModel.Diagnostics.Utility.AllocateByteArray(Int32
            size) --- End of inner exception stack trace
            ---</ExceptionString>
            <InnerException>
              <ExceptionType>System.OutOfMemoryException, mscorlib,
              Version=2.0.0.0, Culture=neutral,
              PublicKeyToken=b77a5c561934e089</ExceptionType>
              <Message>Exception of type
              'System.OutOfMemoryException' was thrown.</Message>
              <StackTrace>at
              System.ServiceModel.Diagnostics.Utility.AllocateByteArray(Int32
              size)</StackTrace>
              <ExceptionString>System.OutOfMemoryException:
              Exception of type 'System.OutOfMemoryException' was
              thrown. at
              System.ServiceModel.Diagnostics.Utility.AllocateByteArray(Int32
              size)</ExceptionString>
            </InnerException>
          </Exception>
        </TraceRecord>
      </DataItem>
    </TraceData>
  </ApplicationData>
</E2ETraceEvent>

Awaiting Nabeel

Answer 1

Are you sure your byte array is only around 2MB? From the exception message that you posted it seems like the message in the buffer is approaching 0.5GB (536870912 bytes). Is this error happening consistently? Are you running multiple service calls at the time this occurs?

Also understand that when using WSHttpBinding with Message Security that the whole message is buffered. My understanding is that this is done to encrypt the whole message in one block so that the corresponding details can be supplied in the header. In other words the header needs the length of the message which is unknown until the encryption is done.

See my related Question where I ask How to best transfer large payloads of data using wsHttp with WCF with message security.