software that store data in flat file, what are common encoding/file format techniques

Question

Hi,

I have been spoiled by either using sql server to store data, or using xml files.

What are common techniques for storing data in flat files other than xml and CSV.

I know many times when I open files that data is all jumbled up, which means it is encoded right?

Are there any common techniques that I could read about somewhere?

Answer 1

YAML. From the site:

YAML is a human friendly data serialization standard for all programming languages.

Example (again, from the site):

Below is an example of an invoice expressed via YAML(tm). Structure is shown through indentation (one or more spaces). Sequence items are denoted by a dash, and key value pairs within a map are separated by a colon.

invoice: 34843
date   : 2001-01-23
bill-to: &id001
    given  : Chris
    family : Dumars
    address:
        lines: |
            458 Walkman Dr.
            Suite #292
        city    : Royal Oak
        state   : MI
        postal  : 48046
ship-to: *id001
product:
    - sku         : BL394D
      quantity    : 4
      description : Basketball
      price       : 450.00
    - sku         : BL4438H
      quantity    : 1
      description : Super Hoop
      price       : 2392.00
tax  : 251.42
total: 4443.52
comments: >
    Late afternoon is best.
    Backup contact is Nancy
    Billsmer @ 338-4338.

Answer 2

A file being all "Jumbled up" doesn't mean it's encoded. it only means that it is saved in some binary format rather then a textual format.
When you're saving in a binary format you simply write the data as it is in the program memory. for instace, to write an integer, you directly write the 4 bytes of the integer and not convert it to text. This has the advantage of saving space.
If you open such a file with notepad all you're going to see is random characters.

Answer 3

Depends on your requirements. If it needs to be human readable then YAML, XML, and CVS are all good.

But if you need performance or storing binary values (say 8 byte double), then personally I create my own file format using Java's excellent DataOutputStream and DataInputStream. But be careful when creating your own format. You generally don't want to share it with the world, or else you will need to care about security and extendability. You'll always have to care about data integrity and maybe add some crc checksums.

Its hard to get custom binary flat files correct, so make sure you truly need one and ask for help!

Answer 4

Out of completeness I'll mention the archaic .ini format, but I can't imagine not writing flat file data in XML or YAML failing that.