I have a case where I need to transfer large amounts of serialized object graphs (via NetDataContractSerializer) using WCF using wsHttp. I'm using message security and would like to continue to do so. Using this setup I would like to transfer serialized object graph which can sometimes approach around 300MB or so but when I try to do so I've started seeing a exception of type System.InsufficientMemoryException appear.
After a little research it appears that by default in WCF that a result to a service call is contained within a single message by default which contains the serialized data and this data is buffered by default on the server until the whole message is completely written. Thus the memory exception is being caused by the fact that the server is running out of memory resources that it is allowed to allocate because that buffer is full. The two main recommendations that I've come across are to use streaming or chunking to solve this problem however it is not clear to me what that involves and whether either solution is possible with my current setup (wsHttp/NetDataContractSerializer/Message Security). So far I understand that to use streaming message security would not work because message encryption and decryption need to work on the whole set of data and not a partial message. Chunking however sounds like it might be possible however it is not clear to me how it would be done with the other constraints that I've listed. If anybody could offer some guidance on what solutions are available and how to go about implementing it I would greatly appreciate it.
I should add that in my case I'm really not worried about interoperability with other clients as we own and control each side of the communication and use the shared interface pattern for data transfered to either side. So I'm open to any idea that fits inside of the constraints of using wsHttp with message security to transfer object graphs serialized using NetDataContractSerializer and I prefer a solution where I don't have to change my existing services and surrounding infrastructure drastically.
Related resources:
- Chunking Channel
- How to: Enable Streaming
- Large attachments over WCF
- Custom Message Encoder
- Another spotting of InsufficientMemoryException
I'm also interested in any type of compression that could be done on this data but it looks like I would probably be best off doing this at the transport level once I can transition into .NET 4.0 so that the client will automatically support the gzip headers if I understand this properly.
UPDATE:
Some history on how I derived at the conclusion that the buffered message being too large was causing my problem. Originally I saw the CommunicationException below while testing.
The underlying connection was closed: The connection was closed unexpectedly.
Eventually after running this and doing some more logging I found the underlying InsufficientMemoryException exception that was causing the problem with the specified message.
Failed to allocate a managed memory buffer of 268435456 bytes. The amount of available memory may be low.
Which originated from the following method.
System.ServiceModel.Diagnostics.Utility.AllocateByteArray(Int32 size)
So in otherwords the failure came from allocating the array. When writing the same data serialized to disk it takes up around 146MB and if I cut it by half then I stop getting the error however I haven't dug much more into the specific threshold that breaks my buffer and whether it specific to my system or not.