Smart or Not: Persist serialized data (dotnet-protobuf, protobuf-net, json) in a Relational DB in CF

Question

I have started reading some of the posts related to protocol buffers. The serialization method seems very appropriate for the transfer of data to and from web servers. Has anyone considered using a method like this to save and retrieve data on the mobile device itself? (i.e. a replacement for a traditional database / orm layer) We currently use a custom reflection-based orm. We want to get away from using reflection on the mobile devices. And, since we have to send/receive serialized data anyway, this seems like a good fit.

• Where would the data be persisted?
• How would the data be queried?

Would it make sense to store the data in a traditional database (SqlCE or SqlLite) with a few "searchable" columns and then one column for the serialized data?

Thoughts? Am I out on a limb here?

Thank you!

Update: this same "theory" could work for other types of serialized data too ... JSON for example. I have been unable to find a NoSQL option for storing and querying serialized data on the Compact Framework. I'd be interested in that option as well if anyone knows of one.

Comment on Object Databases I've tried both db4o and Perst. db4o was absolutely wonderful to work with. I used it in "real life" and the performance, usability, and maintainability were excellent. Their licensing fees for our situation were what I would consider outrageous. Perst was a step down from db4o but also wonderful to work with. It "just worked" and was fast (though not near as nice to query.) Their licensing was very affordable but something in their licensing was unacceptable to the (large, well known) corporation that I contract to. This brings me to where I am now...

Answer 1

Well since no one else has tried to answer this one, I'll toss out my opinion. Bear in mind that I've never used protobuf (which is why I've not answered already), so this is all just based on my loose understanding of things and how, if I had this problem to solve, I would proceed.

First, I've got reservations about sticking blobs into a relational database. This may go back to my bad experiences way, way back in the stone age times of VB6 and may be invalid, but it still gives me pause. As such, I'd probably investigate some other object-storage mechanisms (like Perst or db4o) before trying it.

As due diligence, I'd also at least profile stuffing blobs into a SQLCE database. Why SQLCE instead of SQlLite or some other RDMS? Well, becasue SQLCE supports TableDirect, which I'm a huge fan of. Any time you can access the data without having to use a slow-ass query processor, you're way ahead.

So I'd create a table, give it a couple small look-up columns that I might search on, then an image column holding some large binary blob (the actual blob is not relevent for the test, but it needs to be reasonably close in size to what you expect in production). I'd then add indexes for each of the search columns.

I'd fill the table with a few thousand records and then profile the speed at which I could seek to a desired key (i know that's fast) and more importantly the speed at which I can rehydrate my objects.

I'd then weight the costs (time to develop, opportunity cost, opportunities for reusability etc) of the options and make the decision. I'd probably blog about the results too, as it seems like a problem that would be of a fairly broad interest.

Answer 2

I didn't see this at the time (sorry, but I don't see every question, and while I watch the protobuf-net tag, I don't keep quite as vigilent on protocol-buffers. A very intersting question; In terms of comparison to a document database (rather than relational), I think this type of approach has great potential. It seems a little overkill looking at relational if you are just dropping blobs, but it should work at least. I've certainly used similar setups a lot for data-loading into offline apps - not specifically mobile, but the same concepts apply.

In particular, this type of document-centric approach is useful when you want to rehydrate "the system"; i.e. the state in full. They aren't quite as simple/flexible to query ad-hoc than a relational database. Of course, one option is to rehydrate and they query in memory (for example, in C# the LINQ-to-Objects query over re-hydrated data would be almost indistinguishable from a LINQ-to-SQL query over the data in a relational database).