Safe, Sequential And Scalable Counters In Mnesia

Question

I am writing an application in Erlang/OTP and want to use sequential counters on a version recording system.

I first implemented them with

mnesia:dirty_update_counter

but the experience of using it drove out these hard requirements:

The counters must have the following properties:

• be strictly sequential - 1 followed by 2 followed by 3 etc, etc the sequence is shared across a distributed set of systems and if I have you down as a '3' and you come in a '5' I need to know we have lost some comms and should resync

• safe with a distributed database

mnesia:dirty_update_counter

meets neither of these requirements.

How could I implement a sequential database counter?

Answer 1

Well, since Mnesia data is not guaranteed to replicate atomicly you have to manually lock somehow. I see three alternatives (two with locking and one without):

• Using read locks on the table as you request a new version number. That implies transactions though.

• Use a central server that keeps track of the version numbers and increment them atomically, for example by having a process on one of your nodes which you ask for version numbers.

• Synchronize your servers with NTP and use timestamps.

Answer 2

Have you though about a Message Broker? You could keep it in Erlang by loading up RabbitMQ and setting up a durable queue and persistent messages. Build a consumer that increments a number in the message body by 1 then publishes that new number back to the queue in a new message as it acks the original one.

Not sure it would work for you but it seems like once you manually publish the first message with 1 in the body you would be off and running.

Answer 3

I was going to suggest an erlang process that serves up numbers for you. Since erlang processes messages sequentially you would be guaranteed of getting them in order. Persisting them would be necessary somewhow. The RabbitMQ suggestion above is one way but if you feel like it's overkill then just having the process store to a file periodically should be sufficient. A gen_server should be sufficient for your purposes.

Answer 4

Perhaps the simplest answer is what you are looking for, simply by getting a write lock (for replicated syncing) and making a transactional function that gets the current value of the counter, increments it and saves it back to the DB. Certainly you will get increased overhead as the system scales, but I would recommend writing a server that does this for you and run that on a subset of the nodes, so the lock contention is not directly linked with the cluster size.