How to maxmise the largest contiguous block of memory in the Large Object Heap

Question

The situation is that I am making a WCF call to a remote server which is returns an XML document as a string.

Most of the time this return value is a few K, sometimes a few dozen K, very occasionally a few hundred K, but very rarely it could be several megabytes (first problem is that there is no way for me to know).

It's these rare occasions that are causing grief. I get a stack trace that starts:

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Xml.BufferBuilder.AddBuffer()
   at System.Xml.BufferBuilder.AppendHelper(Char* pSource, Int32 count)
   at System.Xml.BufferBuilder.Append(Char[] value, Int32 start, Int32 count)
   at System.Xml.XmlTextReaderImpl.ParseText()
   at System.Xml.XmlTextReaderImpl.ParseElementContent()
   at System.Xml.XmlTextReaderImpl.Read()
   at System.Xml.XmlTextReader.Read()
   at System.Xml.XmlReader.ReadElementString()
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderMDRQuery.Read2_getMarketDataResponse()
   at Microsoft.Xml.Serialization.GeneratedAssembly.ArrayOfObjectSerializer2.Deserialize(XmlSerializationReader reader)
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)

I've read around and it is because the Large Object Heap is just getting too fragmented, so even preceding the call with a quick check to StringBuilder.EnsureCapacity just causes the OutOfMemoryException to be thrown earlier (and because I'm guessing at what's needed, it might not actually need that much so my check is causing more problems than it is solving). Some opinions are that there's not much I can do about it.

Some of the questions I've asked myself:

• Use less memory - have you checked for leaks? Yes. The memory usage goes up and down, but there's no fundamental growth that guarantees this to happen. Some of the times it fails, it succeeded at that stage previously.
• Transfer smaller amounts Not an option, this is a third party web service over which I have no control (or at least it would take a long time to resolve, in the meantime I still have a problem)
• Can you do something to the LOH to make it less likely to fail? ... now this is most fruitful course. It's a 32-bit process (it has to be for various political, technical and boring reasons) but there's normally hundreds of meg free (multiples of the largest amount for which we've seen failures).
• Can we monitor the LOH? Using perfmon I can track the size of the heaps, but I don't think there's a way to monitor the largest available contiguous block of memory.

Question is: any advice or suggestions for things to try?

Answer 1

You might review the TransferMode property of your binding to see if you meet the requirements to change it from its default value of "Buffered", to "Streamed" or "StreamedResponse".

Also, review the values for maxBufferPoolSize and maxBufferSize. Increasing the size of the internal buffers used can help with memory utilization, especially with processing large messages.

maxReceivedMessageSize is also likely already set if your receiving large messages, but I would review that value as well.

I've seen one of the values above, if your exceeding the threshold, fail with an obscure, memory related message. The original exception was actually hidden by the message that was surfaced to my application. Enabling WCF Tracing helped diagnose the problem and see the real error - I needed to increase the value of one, or more, of the binding properties above.

I didn't get the feel for the binding your using from your post, but I believe that these settings are common across major ones. Check out MSDN documentation on basicHttpBinding for example.

If it is truly LOH fragmentation there isn't anything to be done about it once tuning efforts have been exhausted. Rolling recycle of the application might be required to mitigate it (I hate recommending that) but if you've exhausted other efforts you might be left with that.

Answer 2

While I haven't tried this and suppose it can be hard, I believe it is possible to analyze the LOH and see what the largest available continuous block of memory is, and which types of objects are on the LOH. You can do this by making a dump of the process multiple times and analyzing it with WinDBG + SOS. It could be cumbersome, because WinDBG will give you memory locations of objects, but perhaps it contains some operations that make it easier.

Answer 3

I think your problem Might be an Assembly Leak caused by using XmlSerializer and not using one of two constructors as indicated in this MSDN article:

To increase performance, the XML serialization infrastructure dynamically generates assemblies to serialize and deserialize specified types. The infrastructure finds and reuses those assemblies. This behavior occurs only when using the following constructors:

XmlSerializer.XmlSerializer(Type)

XmlSerializer.XmlSerializer(Type, String)

If you use any of the other constructors, multiple versions of the same assembly are generated and never unloaded, which results in a memory leak and poor performance.

Nice, huh. The answer is to cache your XmlSerializer (assuming you even create it).

To really figure it out you need to do what Tess tells you to do. She's a freakin genius.