Handling massive in memory objects in .NET

Question

hey guys, here's the issue,

Essentially, I have a very large List containing in turn relatively large Dictionaries

So basically, I have a very big in memory collection.

I then serialize this collection manually to XML, and send it over http. needless to say, the XML is too large, sometimes so large I get an OutOfMemory exception before even trying to send it.

In .NET, how would I go about calculating potential memory usage. For example, in this case, I have to break down the XML into chunks, by processing only small amounts of the Collection at a time.

How do I efficiently calculate the size of each "chunk" on-the-fly. I don't want to pick an arbitrary number like, "process 100 items at any one time", I want to know, approximately, how big each chunk should be for a case by case basis.

cheers

UPDATE

Although @Jacob provided the best solution for this particular problem, the conceptual structure of the app is itself flawed.

Indeed, the solution is to execute a fraction of your message, in order to calculate how potentially big the message will be, when working with a collection.

You then send each acceptably sized unit, one by one.

But this is only a hack. The real solution is to either find a way to not pass large messages, or use a completely different protocol altogether.

There's an interesting post on the subject here though if you want to use SOAP, however I decided to find a way around sending so much data.

Answer 1

Why don't you just stream the data so you just convert to XML on the fly avoiding having a huge XML file in memory?

Answer 2

How are you sending it? You should do so through WCF, which can do streaming. It would also give you a choice, through configuration, of whether to use XML or binary, or whatever.

Answer 3

I think you might have a conceptual problem more than anything else. "Calculate potential memory usage" is at odds with "efficiently calculate the size of each chunk". The only way to really get at your memory usage to the degree of accuracy where you can predict adequate chunk size is to actually make your conversion.

It sounds like the best way to come at this efficiently might be to tackle it progressively--essentially what those who are suggesting streaming objects are saying. If you can't leverage actual streaming, you'll probably want to structure your serialization so that you progress one conceptual unit at a time (i.e. one item in your list with it's attendant dictionary children).

Answer 4

If it is a problem to send, isn't it also a problem to receive? You sound like you're trying to solve half a problem. XML is a big no-no for large data.