Online mnesia recovery from network partition

Question

Is it possible to recover from a network partition in an mnesia cluster without restarting any of the nodes involved? If so, how does one go about it?

I'm interested specifically in knowing:

• How this can be done with the standard OTP mnesia (v4.4.7)
• What custom code if any one needs to write to make this happen (e.g. subscribe to mnesia running_paritioned_network events, determine a new master, merge records from non-master to master, force load table from the new master, clear running parititioned network event -- example code would be greatly appreciated).
• Or, that mnesia categorically does not support online recovery and requires that the node(s) that are part of the non-master partition be restarted.

While I appreciate the pointers to general distributed systems theory, in this question I am interested in erlang/OTP mnesia only.

Answer 1

It works like this. Imagine the sky full of birds. Take pictures until you got all the birds. Place the pictures on the table. Map pictures over each other. So you see every bird one time. Do you se every bird? Ok. Then you know, at that time. The system was stable. Record what all the birds sounds like(messages) and take some more pictures. Then repeat.

If you have a node split. Go back to the latest common stable snapshot. And try** to replay what append after that. :)

It's better described in "Distributed Snapshots: Determining Global States of Distributed Systems" K. MANI CHANDY and LESLIE LAMPORT

** I think there are a problem deciding who's clock to go after when trying to replay what happend

Answer 2

Sara's answer is great, even look at article about CAP. Mnesia developers sacrifice P for CA. If you need P, then you should choice what of CAP you want sacrifice and than choice another storage. For example CouchDB (sacrifice C) or Scalaris (sacrifice A).

Answer 3

After some experimentation I've discovered the following:

• Mnesia considered the network to be partitioned if between two nodes there is a node disconnect and a reconnect without an mnesia restart.
• This is true even if no Mnesia read/write operations occur during the time of the disconnection.
• Mnesia itself must be restarted in order to clear the partitioned network event - you cannot force_load_table after the network is partitioned.
• Only Mnesia needs to be restarted in order to clear the network partitioned event. You don't need to restart the entire node.
• Mnesia resolves the network partitioning by having the newly restarted Mnesia node overwrite its table data with data from another Mnesia node (the startup table load algorithm).
• Generally nodes will copy tables from the node that's been up the longest (this was the behaviour I saw, I haven't verified that this explicitly coded for and not a side-effect of something else). If you disconnect a node from a cluster, make writes in both partitions (the disconnected node and its old peers), shutdown all nodes and start them all back up again starting the disconnected node first, the disconnected node will be considered the master and its data will overwrite all the other nodes. There is no table comparison/checksumming/quorum behaviour.

So to answer my question, one can perform semi online recovery by executing mnesia:stop(), mnesia:start() on the nodes in the partition whose data you decide to discard (which I'll call the losing partition). Executing the mnesia:start() call will cause the node to contact the nodes on the other side of the partition. If you have more than one node in the losing partition, you may want to set the master nodes for table loading to nodes in the winning partition - otherwise I think there is a chance it will load tables from another node in the losing partition and thus return to the partitioned network state.

Unfortunately mnesia provides no support for merging/reconciling table contents during the startup table load phase, nor does it provide for going back into the table load phase once started.

A merge phase would be suitable for ejabberd in particular as the node would still have user connections and thus know which user records it owns/should be the most up-to-date for (assuming one user conneciton per cluster). If a merge phase existed, the node could filter userdata tables, save all records for connected users, load tables as per usual and then write the saved records back to the mnesia cluster.