WCF - Memory problems when deserializing a large streamed objects

Question

Here's my situation - I have a simple object model. At the root, there's a Report object. Reports have an array of Documents. Each document has an array of Chapters and Page. Each of these objects has 4 or 5 properties (primitives - strings and integers). It looks something like this:

• Report
  • List< Document>
    • List < Chapter>
    • List < Page>

Simple enough, right?

Here's my problem. After a long-running process completes, I have a report object that generally has about 5000 Document objects. Each Document has roughly 200 Chapter objects and 200 Page objects.

In other words, the Report object has a rather large memory footprint. 5,000 documents, 1,00,000 Chapters, and 1,000,000 Pages. If I serialize the object into a memory stream, it can be as large as 80MB.

The problem is that I have to move this object across the wire (yes, that's a MUST).

That's where I need help!

My first attempt was simply to try to move the entire object across the wire using a standard WCF service with the object model decorated with DataContract/DataMember attributes. Given the number of objects in the graph, and the overall size of the serialized object (since WCF is creating XML), it's not feasible. Even setting the maxReceivedMessageSize and maxItemsInObjectGraph to their highest possible values isn't getting the job done. Plus, that's a buffered request and that just doesn't seem like a good idea.

My second attempt was to create a WCF StreamedRequest using a System.ServiceModel.Channels.Message instance. That seems to work to get the large Report object across the wire without any issues. The request is streamed, so the transport isn't an issue. The problem, though, is that I need to deserialize the object when the WCF service receives it. I've tried a couple different approaches (all stream-based) with a consistent problem. Deserializing the object takes the W3WP's memory from 115,000K (normal) to over 2.2GB! And that's deserializing just 1 Report object. Plus deserialization is extremely slow!

I don't know if the 2.2GB spike in memory is because of the WCF settings for a property like maxReceivedMessageSize or if there's some leak in the deserialization process (I'm currently using the DataContractSerializer). Whatever it is, I'm stuck.

So, there you have it - a large object needs to go across the wire and then be deserialized without taking an eternity and consuming all the available memory.

Does anyone have any suggestions? I'm totally open to just about any solution; I don't care how unusual or creative it is. The only limitations are:

1. Needs to be hosted in IIS
2. Needs to be reasonably efficient with memory consumption.

Thanks, everyone!

Answer 1

I'd break it into multiple calls. The first call is a request to create the report and persist it, and return an object that identifies the report. The subsequent calls are to request a reasonable piece of the report by passing the report object to another method. Maybe request a range of documents.

Unless the consumer really need all 5000 documents loaded into memory at once (which sounds bad), then you should be able to chunk through the parts.