serializing and deserializing large files

Question

Is there any good approach to serialzing and de-serializing large files (>10M) in c#.

Thanks in advance.

Answer 1

You can check out my answer here to this question (there are all kinds of other relevant answers there too).

My method uses BinaryReader and BinaryWriter for performance.

I have used this method to deserialize 50MB files in a recent project, and it does it quite quickly (under 5 seconds) compared to the built-in serialization or Xml serialization (10 minutes for my data set).

Answer 2

There isn't any difference between de/serializing small or large files. You would just have to make sure that you don't deserialize very large files to memory, that's going to buy you OOM.

And large files are going to take more time of course. If that makes your user interface unresponsive then you'll want to do this processing in a background thread. BackgroundWorker is a typical solution for that.

Random shots in the dark here btw, your question is far too vague.

Answer 3

If you really have large files (let's say larger than 100 MB), the best thing is to load only the things you need at the moment.

Let's say you have a list of 10.000 customers - each with an image. It makes no sense to keep this list in the memory.

For example, you could load all lastnames and the position of the person in the file. So the user could search for a person and you could load exactly that person.

Another possibility would be loading the first ten and display them to the user. As soon as he clicks at a "Next" button you could load the next ten - just plan how to organize the information.

Instead of very large files, databases can bring some advantages. They can abstract the large amout of work required to navigate within the file.

"Single-Line-Serialization" using BinaryFormatter etc., however, reaches its limits at files of that size in my opinion. You have to think of other concepts.

Answer 4

Are you sure serialization/deserialization is the right approach for that much data? Would perhaps a client-side database, like SQLite be a better solution, where you can query it for the exact data you need, instead of just loading everything into memory?