Recommended techniques for field updating embedded Linux safely

Question

Embedded Linux based devices often require a mechanism to update applications and system files. For example, a (non-networked) lab instrument with a USB port can get software updates from a USB stick.

It would be a simple matter to run a script to copy files into place on the device's internal flash memory. However, there is the danger that the device would lose power in the middle of the update, and end up a brick.

The situation for application files is a bit easier since there is room to duplicate the application directory, update one copy, and quickly swap old and new directories minimizing the failure window.

Things are dicier for kernel and system files since they are spread out throughout the file system.

We have used hard and soft links in the file system to identify critical files. We use hashes on files and archives to verify file integrity. We have considered using an emergency ramfs in the kernel to provide a fallback if starting from the updated file system fails.

What are your approaches to this requirement?

Answer 1

If you must ensure the reliability, you can have two flash partitions (or even chips), one with the current working configuration and one with the new configuration. Then use a hardware watchdog which will reset the unit and switches the active boot flash partition to the "last known good" configuration.

Answer 2

I would go with the same approach as with the application files: Make for the critical files and complete own partition, link to them, and duplicate the partition. In all of your init you should as first check if the links show all to the same partition, if not, reset them (to the partition with the files with the newest date of a certain file). If you want to update just copy everything onto the new partition, and if everything is fine (crcs ok) loop over the files and set for each the link from one filesystem to the other.

This way your critical files should be always be in sane state.

Scenarios:

1. Update fails while copying files onto new partition

   No Problem because the links show still to the old working ones.

2. Update fails while linking

   No Problem because all new files are valid and already copied (else the relink step wouldnt have start), setup check correct this

Answer 3

Have at least two partitions. I'd suggest 4

• boot

• alternate boot

• program data backup

• program volatile data

Use grub fallback booting to boot alternate if boot fails.

So if the update fails, the alternate works.

NEVER update the boot loader.

If the data partition is toasted, reformat and copy over the backup data partition.

Now you can't fail unless the flash disk dies. If you are using COTS hardware, and main disk was say, Compact flash, you could have a physically isolated backup on say, a little USB key.