Light-weight flexible data persistence options

Question

I was originally using SQLCE first starting with LINQ to SQL and then moved to Entity Framework, but it seems that SQLCE is too inflexible. (Right now I can't even drop a table to recreate it because it says I can't remove PrimaryKeyA because it is being referenced by PrimaryKeyA... so I don't have much confidence in being able to modify it later)

I need flexibility in that the app is going to be released over many iterations and data needs to be able to change fairly easily. Newer versions of the app need to be able to read data from older versions, either directly or through a conversion.

It needs to be light-weight to deploy, under 10MB, but preferable under 5. The user will (should) not be modifying the data outside the app, and they don't need to know or care how the data's stored.

High performance isn't an issue, it's just going to be a single user's data. Right now I'm tempted to say just go with XML to leverage LINQ to XML. But I'm wondering if there's any other suggestions. I'd like something that's easily translatable to and from normal objects, would XML serialization work for this?

I guess what I'm looking for is an easy way to persist objects that can be easily updated to new versions with object changes. Performance isn't a huge issue but it should be at least tolerable for a small/medium amount of data. (Basically an electronic form of a cook book for a single user's recipes)

Answer 1

There's always binary serialization to a file. It's not sexy, but it works, and it doesn't require any third party tools or software. Just make sure you precede every data grouping with a version number (usually referred to as a Schema number). That will alleviate headaches down the road when you change the data model.

In an OOP setup typically every object that needs to be persisted has a Serialize() method that takes a direction parameter (saving vs. loading) and an object that represents the binary file. When writing out you just need to make sure that every object gets serialized out. When reading in you need to create the objects as you go. This process is often made easier by a container that knows how to serialize the object in and out.

Conversion is handled on the fly by the various Serialize() methods. For example, if an object that is currently on Schema version 5 encounters data that was written using Schema version 4 it will know how to handle that.

However, if you need SQL like querying capabilities this may not be your best option. This works best when you are going to read in all of the data.

EXAMPLE:

Say you a class Foo that has 2 member variables you want to serialize:

class Foo
{
public:
  const unsigned int SCHEMA = 1;
  int i;
  double d;

  void Serialize(bool bSaving, CBinaryFile file)
  {
    if (bSaving)
    {
      // Serialize everything out
      file << SCHEMA << i << d;
    }
    else
    {
      // Read in the schema number first
      unsigned int nSchema;
      file >> nSchema;

      // Validate the schema number
      if (nSchema > SCHEMA)
      {
        // We're reading in data that was written with a newer version of the program
        // Since we don't know how to handle that let's error out
        throw exception;
      }

      // Read everything in
      file >> i >> d;
    }
  }
}

Now let's say in a year you add another member to Foo. You would handle it like this:

class Foo
{
public:
  const unsigned int SCHEMA = 2;
  int i;
  double d;
  string s;

  void Serialize(bool bSaving, CBinaryFile file)
  {
    if (bSaving)
    {
      // Serialize everything out
      file << SCHEMA << i << d << s;
    }
    else
    {
      // Read in the schema number first
      unsigned int nSchema;
      file >> nSchema;

      // Validate the schema number
      if (nSchema > SCHEMA)
      {
        // We're reading in data that was written with a newer version of the program
        // Since we don't know how to handle that let's error out
        throw exception;
      }

      // Read everything in
      file >> i >> d;
      if (nSchema > 1)
        file >> s;
    }
  }
}

As long as you wrap everything in Schema numbers on the fly conversion is straight forward.

Answer 2

I suggest using System.Data.SQLite. It gives you straightforward, high-performance SQL through the ADO.net interfaces and it's lightweight. It's also fairly well documented and easy to troubleshoot using the command line sqlite.exe client along with other tools. One significant upside to SQLite is that it's got relatively loose typing, so you don't have to spend much time messing with schemas.

I can't speak to whether it has any support for LINQ, but it's quite good.

Answer 3

I would suggest that XML route you are considering might be a reasonable idea. While more verbose than binary serialization, it's more easily debuggable if need be; and compresses nicely (on-the-fly) if less disk usage is needed.