Delphi Binary/Text File Necessity

Question

I am an engineer and not a software programmer, so please excuse my ignorance.

I have written a Delphi(7SE) program to read “real” datatype from a USB port connected to two digital thermometers.

I have completed this much of the program.

What I have not completed as yet is explained by the following:

I wish to save this “real” data to a Binary File(s). A text file would be fine as well, but i'm concerned about having a big data file.

I also wish to read this data back from the Binary/Text File to display the data using my Delphi application.

I don’t think this would be too difficult. I currently save my data in .CSV format.

The twist here is that the binary file should contain data from different sessions initiated by the user of my application.

So when I click on say, a button called “ historical” data, a new window/form would pop up that would show different session times that I had started & stopped from earlier times. Then a session would be selected and data then retrieved for displaying.

Can this be done in one binary files or would you have to use 2 files: one for the “real” data and another which indexes the different session times?

My requirement for this way of saving binary data is that I would not have to keep typing in filenames and therefore keeping track of many data files.

For example a thermo.hst(historical data) and a thermo.idx (index file) file would contain all the information such as actual temp data, time of read data, session start & end times etc.

Any useful pointers and hopefully code with as much detail would be greatly appreciated.

Answer 1

The data can be interleaved. Just start every block (a set of history) with a header that identifies the block and contains its length. When reading you can then easily separate that.

You can hold an additional index file next to this for fast access if you require, but if this is the case, I would start studying some embedded database. (TDBF, sqlite or embedded firebird).

I would also head in the database direction if I expected that my querying would get more complicated in the future.

If it is all about logging, the data doesn't get gigantic and the performance of the view is fine, I would keep the binary file and avoid the hassle of having users to install and maintain a databsae solution. (TDBF is maybe an exception to that, since it is completely statically linked)

Answer 2

By using a database, you've in part just renamed part of the problem from "typing in file names and keeping track of many data files" to "typing in data set name and keeping track of many data sets."

In both cases, as an example, either the user or the program has to create a new file/data set name, and choose from a list of files/data sets to open later. And in both cases you have to make a call to a function named something like "DeleteDataSet".

You might re-consider whether you really need a database (with associated learning curve for API, how to define the data structure, update it in the field when something changes, browse the data in a viewer, access the data programatically, proprietary format, database maintenance, repair tools, installation, etc.) I'm sure that you could learn all these, and they might be valuable in future projects. But, maybe a simpler approach would be more appropriate and adequate for this one-time project by a non-software engineer.

If you're willing to have a proliferation of many unique, standalone data files on one folder, I'd encourage you to stick with what's working: use one CSV file per data set. (Have you run into speed or size issues with CSV files containing a single data set thus far? That would be an enormous amount of data!) One nice thing about CSV files is that you can just pop them into an editor to view or edit...

And, then, add a second file that contains filenames, and other descriptive information. This file would be a simple TIniFile:

[My Name one]
Date=06 June 2010
StartTime=12:30pm
StopTime=3:15pm
FileName=Data1.csv
[My Name two]
...

The tools available in Delphi for TIniFile will let you easily manage this list, including ReadSections into a string list that you can just assign to a combo box for the user to select a data set. (See example below) And, like the CSV files, you can just edit the .ini file in any text editor.

You'll need to build a UI to allow a user to delete a dataset (section in the ini file and associated .csv file). To give you an idea how the ini file would be used, here's the pseudo-code for deleting a data set:

(In IDE Object Inspector, set ComboBox.Style := csDropDownList to prevent user from typing in a name that doesn't exist.)

Load a combo-box that shows available data sets.

1. ComboBox.Items := IniFile.ReadSections;

In the combo-box's OnSelect event handler:

2. DeleteFile(IniFile.ReadString(CombBox.Text, 'FileName', '')); 
3. IniFile.EraseSection(ComboBox.Text); // remove the section from the inifile

Heck, that's not a lot of code, even after you add a bit of protection and error checking!

Maybe the above solution will be voted down by others here as trying to put a round peg in a square hole or re-inventing the wheel. And I might agree with them. There are good arguments against this approach, including proliferation of many files. But, if it was me, I'd at least consider this approach as keeping-it-simple and not requiring anything new but that you learn the TIniFile object, which is quite powerful.

Answer 3

I hope this sample code isn't too late to be helpful.

(I've added this as another answer from me so that I can cleanly list the code. If this or my previous post answers your question, please click the answer icon so I get reputation points!)

Below is some rough code that shows how to read the sections in an ini file and find the largest filename. I confirmed it compiles and seems to return valid values, but you'll need confirm it does what you need. It's more to show you the idea...

Note that if your data filenames have an extension, you'll have add code to remove the extension in my sample code using something like: FileName := ChangeFileExt(Filename, '').

  // Call with an open inifile.  Returns the name of the next filename, or '' if trouble
  Function GetNextFileName( const IniFile: TInifile):String;
  const
    BASE_FILENAME = 'File.'; // sections in the ini file will be [File.1], [File.2], ... [File.100], etc.
  var
    Sections: TStringList;
    NumericPartAsString: String;
    NumericPartAsInteger: Integer;
    ListIndex: Integer;
    LargestFileNumberSeenSoFar: Integer;
  begin
    Result := '';
    Sections := TStringList.Create;
    IniFile.ReadSections(Sections); // fills StringList with the names of all sections in the ini file
    if( Sections.Count = 0) then
      Result := BASE_FILENAME + '1'
    else
      begin  // find largest extension
        LargestFileNumberSeenSoFar := -1;
        ListIndex := 0;
        while ListIndex <= (Sections.Count - 1) do  // for every string (which is also a filename) in the string list:
          begin
            NumericPartAsString := StringReplace(Sections.Strings[ListIndex], BASE_FILENAME, '', []); // remove base filename
            if (NumericPartAsString <> '') then
              begin
                NumericPartAsInteger := StrToIntDef(NumericPartAsString, -1);
                if (NumericPartAsInteger > LargestFileNumberSeenSoFar) then
                  LargestFileNumberSeenSoFar := NumericPartAsInteger;
              end;
            inc(ListIndex);
          end;
        if (LargestFileNumberSeenSoFar > -1) then
          Result := BASE_FILENAME + IntToStr(LargestFileNumberSeenSoFar + 1);
      end;
    Sections.Free;
  end; { GetNextFileName }

  procedure TForm1.Button1Click(Sender: TObject);
  var
    IniFile: TInifile;
    NewFileName: String;
  begin
    IniFile := TInifile.Create('c:\junk\ini.ini');
    NewFileName := GetNextFileName(Inifile);
    if (NewFileName = '') then
      ShowMessage('Error finding new filename')
    else
      ShowMessage('New filename is ' + NewFileName);
    IniFile.Free;
  end;