Blackbox type data logging

Question

In a Linux embedded application I'm developing, there is the need to record some events that happen from time to time. These records are saved on a MTD flash device and once written there is no need to change them or do efficient searches, but read access is required to show the data back to the user. A big problem is that power can go away at any time, without a proper shutdown sequence. The frequency these events occur can be very slow (days/weeks), but several of them will occur at once. The data to be saved for each event is strongly typed: date, time, a couple of short text strings and several integers.

Currently I inherited a solution based on jffs2 and SQLite that is far from optimal because the DB file sometimes get corrupted. When this happens the whole file gets unreadable and there is no way to understand if it was caused by a bug in jffs2, or in SQLite or if the flash sector was bad, or if the power was cut at the wrong time.

Is there a library or a combination of filesystem/library that can better help me solve this kind of problem ? Or should I just use a text file with a CSV-like format ?

Answer 1

I'm not expert on embeded systems, but I would think that a CSV would probably be best. It basically can't be corrupted, or if it does, then you can easily see the error and fix it manually (new line or just removing a line). I have been working on receiving the data from an embeded system where they have a lot of corruption problems (partially on the system and partially during the phone line transfer). It would be very helpful if it were in a CSV type format so we could find the errors and remove or fix them instead of corrupting the entire data set.

If you aren't needing to search within the system, then a CSV works perfectly.

Answer 2

There is a number of embedded file systems (not fat compatible) that designed exactly for this purpose. I can't suggest since never used one, but here something from google. I'm sure you can dig more, and hopefully somebody here can provide more info, may be there is something GPL based. Comparison of different file systems are here

Answer 3

Two csv/text files. Start a new pair each time the system restarts. Write each event to the first file, flush the file to store, write the record to the second file, then flush again.

This way, if you crash during the first write all the data in the second copy (up until that write) will still be there.

Make sure the flush is a full file system flush and not just the clib buffer flush.

Maybe also place the files on separate file systems. Reserving space ahead of what you need could also help speed up the process.

Answer 4

We are using plain old syslogd to a YAFFS2 partition on NAND flash, it appears to work well: when messages are sent to the logger and power is removed immediately after (<100ms) the message is there and the log never appears to corrupt.

This is based on observation rather than my explicitly knowing that everything will always be consistent by design, mind.

Answer 5

What facilities are available to you? The best option is often to log to an external resource, for example via syslog, SNMP, raw socket, or serial port. This protects you logs from unpleasantries on the device itself.

If you need to store logs internally, I've found plaintext, human-readable files to be the best option in embedded devices. The "write/flush" cycle is fast, no tools are needed to maintain them, and you can monitor them in real-time. If file size is a problem, you can timestamp with an integer rather than formatted text, and you can use a numeric "Event ID" to abbreviate each log (leave only the instance-specific data as text).