Array serialization performance issue

Question

In my Windows Mobile (.NET Compact Framework) application I use a big array to store the application's primary data. This is a dataset of potentially hundreds of objects. Each object has about 10 or so properties and two arrays of itself, each with about 25 other objects, each of which has about 5 properties.

To save this array on the mobile device, I simply serialize the entire array. This works well most of the time and it's very, very easy.

However in our test cases we always worked with only a handful of objects, a maximum of about 50 to 75. But our client has had cases where users had several hundreds of these objects, up to 1000. In those cases the serialization is quite slow, it can take up to a minute.

The actual problem is when saving the entire array, mostly only a couple of the objects have actually changed. So the basic flow is something like this:

• Load the entire array from storage, say 400 objects;
• Change a few properties of 1 object;
• Save the entire array back to storage, the full 400 objects;
• Change a couple more properties of that same object;
• Save again
• Change the final properties;
• Save again;
• Same with any subsequent objects...

It wouldn't be a problem normally if saving didn't occur too often, but on several intermediate steps the data is saved. This is to be sure all data is persisted and no data loss can occur (for example when the battery dies).

How can I solve this?

Answer 1

So to be clear, le't make sure I understand your scenario:

• What you have is some form of serialized array (you've not stated the format as XML, binary or other) as your data store?
• And if one property changes, you rewrite the entire array, even if there are 1000 objects with sub-objects?
• And you're writing to Flash, not just RAM?
• And for a full "save" you're doign the write operation a few times?
• And for some reason, you're finding that this is slow and it gets slower the larger the data set gets?

The answer is actually fairly simple. This is completely expected behavior based on how you're doing this. Why would you use this mechanism for a data store, especially for large, frequently changing items? This is a classic example of a poor design decision. When a property changes, you should be changing just that property in the store and serialized arrays simply are not well suited to this.

You should be using an actual database engine, whether it's an RDBMS or an object database, but something that's doing way, way less writing to the storage medium. If you need the data as an array for transfer to the PC/Server, that's fine - create a mechanism to extract from the store and put it into an array for that purpose.