How to store an object to disk?

Question

How to store an object to disk in all its glory? My object is derived from TObjectList so it holds other objects.

Which is the fastest and easiest way? Which is the compatible way?

Serialization IS NOT a solution since I want to save also non-public properties and the list of objects it holds!

For the moment I trying to save every object independently as a binary file which then are packed together. This is a lengthy process but allows me to load old versions of the object with a newer version of the program (compatibility with previous saved projects). Anyway, the complexity started to grow and it is not looking good anymore.

Answer 1

You need to come up with some way of encoding that object as a string so that when you construct a new object with that string you get the same object. This is called serialization, and some languages provide that functionality for you.

For example if you have this object:

class serialize_me {
 private:
   int a;
   float b;
 public:
   double c;
}

and a = 5, b = 3.2, and c = 67.5

your string might look like this:

a5b3.2c67.5

Then you could parse that string and assign the appropriate values to all the members.

I think it goes without saying that strings are easy to store on disk.

EDIT: This is a very basic example, but I think you can understand the concept easily enough.

EDIT: Delphi specific serialization. At the bottom of the page there is a link to a full-fledged XML serializer class.

Answer 2

Lots of ways of doing this I guess, in the past I have used inifiles for this purpose, useful as TMemInifile is there for you to do a lot of grunt work in memory before you save out to disk. Since you have a hierarchy to save out you might want to consider using something like XML, but I haven't done this personally so cannot advise on that one.

Answer 3

If the object is derived from TPersistent, XML or JSON serialization are easy to do with open source libraries:

• OmniXML
• SuperObject

One way to make versioning easier is an anti-corruption layer between the domain model and the persistence layer. (maybe using data transfer objects which will not change with every change of the domain model).

For automatic versioning see this article: Migrate Serialized Java Objects with XStream and XMT

XMT introduces class VersionedDocument to version serialized XMLs and handle the migration. The same design could easily be implemented with Delphi.

Answer 4

If you are using Delphi 2010, things will be much easier because of the new RTTI unit, Robert Love has written a nice unit to serialize the objects to XML called XMLSerial.

You can read about it at his blog : Xml Serialization - Basic Usage

Answer 5

I also mostly use handcrafted serialization for my own datastructures. The multiple version angle is one of the main reasons.

However in your case that is difficult, since not all your objects (tobjectlist) derive from an own hierarchy that contain virtual abstract methods to load/store.

D2010 serialization (which afaik allows nearly everything to RTTI) could be a solution, but probably requires a new delphi version, and worse, it spells the end to manually dealing with versioning. (e.g. copying values from old fields into new ones when the format changes)

If the manual streaming is getting out of hand, a different approach could be to have abstract definitions for the data part of your objects, and generate sourcecode (the field declarations and the streaming code) from these abstract definitions. The advantage is that you might be able to slip in some custom code here and there when you need, or patch your generator for versioning issues.

I did this once for a business object to SQL mapping with over 800 objects. Since it was the time before generics in Delphi, I generated a typesafe container type for each object too, as well as other helper and converter objects/routines.

It is a lot of work to setup though, and only is worth it if you have a project with really a lot of objects and fields (hundreds if not thousands) and are sure that you will need to maintain it with significant mutations for quite some time to come.

Answer 6

You state that serialization is not a solution but I ask why not? I've done something like this in the past but here is what I've done.

I created a component class that did nothing but serialize a non-TPersistant based object so that I could stream it in and out using VCL streaming capabilities.

For example:

//Please forgive me for any errors that exist, as I'm trying to type this from the top of my head. As well, this is not going to be functionally complete.

unit streamlist1;

interface

uses MyListObjectUnit;

procedure SaveList(fielname:string; data:TMyListObject);
procedure LoadList(filename:string; var data:TMyListObject);

implementation

type
  TMyListStreamer = class(TComponent)
  private
    fMyList : TMyListObject;
    procedure ReadList(Reader:TReader); //This is where the magic happens
    procedure WriteList(Writer: TWriter); //This is where the magic happens (x2)
  public
    procedure DefineProperties(Filer: TFiler); override; //defined in TPersistent
    procedure AssignMyList(data:TMyListObject);
    procedure PopulateData(var data:TMyListObject);
  end;


TMyListStreamer.procedure DefineProperties(Filer: TFiler); override; //defined in TPersistent
begin
  Filer.DefineProperty('MyObjList', ReadList, WriteList, true);
  //Filer.DefineBinaryProperty('MyObjList', ReadList, WriteList, true); //your choice
end;

procedure TMyListStreamer.ReadList(Reader:TReader); //This is where the magic happens
begin
  //Use the reader class to read in anything you want...
end;

procedure TMyListStreamer.WriteList(Writer: TWriter); //This is where the magic happens (x2)
begin
  //Use the writer class to write out anything you want...
end;

procedure SaveList(fielname:string; data:TMyListObject);
var
  wFile : TFileStream;
  wList : TMyListStreamer;
begin
  RegisterClass(TMyListStreamer);
  Try
    wFile := TFileStream.Create(filename, fmcreate);
    wList := TMyListStreamer.create(nil);
    try
      wList.AssignMyList(Data);
      wFile.WriteComponent(wList);
    finally
      wFile.Free;
      wList.free;
    end;
  finally
    Unregisterclass(TMyListStreamer);
  end;
end;

procedure LoadList(filename:string; var data:TMyListObject);
var
  wFile : TFileStream;
  wList : TMyListStreamer;
begin
  RegisterClass(TMyListStreamer);
  Try
    wFile := TFileStream.Create(filename, fmOpenRead);
    try
      wList := TMyListStreamer(wFile.ReadComponent(Nil));

      if assigned(data) and assigned(wList) then
        wList.PopulateData(data);

      if assigned(wList) then
        wList.free; 
    finally
      wFile.Free;
    end;
  finally
    Unregisterclass(TMyListStreamer);
  end;
end;

Using this method you can stream (serialize) anything out of the VCL or custom data. It takes a little to set up, but the strength is that you can control everything going in and out of the data file. You can even, with a little fore thought, create a version flag and process different data by ignoring or massaging specific data in newer versions of the program/component.

You can even stream other VCL objects out of your streaming component as long as you already know the type of object (ie TComponent/TPersistant based objects) by using the existing methods of the TReader/TWriter.

Not a full solution but it should get you where you want to go with a little more work.

Answer 7

Your current solution is probably the best if you're going to have separate versions of the object over the years.

What I do is create SaveToStream() and LoadFromStream() methods, and manually write the object's properties to a tstream in a fixed order, prefixing it with a version number of the structure. The benefit of this, as you mention, is that you can better adapt to older versions of the stream. For example, if you have 5 versions, but you need to initialize something in a certain way for version 3 files, its easily done. You then wrap a SaveToFile() function around it that creates a TFileStream and calls SaveToStream().

I believe there's a TWriter class that lets you write various datatypes to the stream more easily... or you can create your own simply enough. (I made my own filestream descendent to handle this)

If saving multiple objects to a single stream, you may want to note the position before each object gets written, and then go back and mark a length, so that you (or someone accessing the files) can skip ahead through the file without reading it.

Also, if you have a hierarchy of classes that you want to save, 'bottom load' the ancester class with all the properties you want to save to a file. This way you only need one implementation of the save routine. Its a little less efficient since you're carrying around variables you dont necessarily need in all the objects, but its far simpler to manage.