Distributed Erlang and the CAP theorem

Question

By the CAP theorem, it is impossible for a distributed Erlang system to simultaneously provide all three of the following guarantees:

• Consistency (all Erlang runtimes, or nodes, see the same data at the same time)
• Availability (node failures do not prevent survivors from continuing to operate)
• Partition Tolerance (the system continues to operate despite arbitrary message loss)

A distributed Erlang system can support zero, one, or two of the guarantees.

Using Erlang and OTP, how can each guarantee be implemented? Most distributed Erlang applications make the practical choice for higher levels of A and P, and settle for "eventual consistency". It seems Erlang itself was designed to support distributed (P), fault-tolerant (A), soft-real-time, non-stop applications.

The programming language (Erlang), run-time system (ERTS), and set of libraries (OTP) are designed for building distributed fault-tolerant applications; how do I do the three things that define distributed fault-tolerant applications?

Answer 1

EDITED for clarity

It depends on the design of your application and not the platform in which you implement it. You could implement any 2 of the guarantees, with just about any language or platform combination.

Answer 2

A really good example I found is Riak, a distributed key-value store based on Amazon's Dynamo and built using Erlang and OTP. The tunable CAP controls let you dynamically "choose" your levels of consistency and availability; it chooses to focus on the A and P of CAP, which makes it eventually consistent. Riak is open-source, and there are a number of good articles online regarding its implementation.

Another good example is Dynomite, PowerSet's Dynamo-clone-in-Erlang. Although the project has been dead for a while, it serves as a useful functional design document on how to build a Dynamo clone.

These two projects are themselves based on Amazon's Dynamo, an incrementally scalable, highly-available key-value storage system. The technology is designed to give its users the ability to trade-off cost, consistency, durability and performance, while maintaining high-availability. The paper is a good read.

Answer 3

I think this theorem should be considered in only one layer. What is a layer? It is something like OSI-ISO layers but for databases: application/db, OS, disk/hardware. All of the CAP conditions cannot be satisfied in one layer. Erlang/OTP operates in application layer so cover 2 of them with Erlang (you can choose, because of flexibility of Erlang) and the last one cover with OS (i.e. file system) or hardware layer (raid).

Answer 4

Erlangs strength should be with chosing A&P, but as with any system it is possible to chose any two or less. This is partly due to erlangs programming model with all communication through message passing. If you use good Erlang style you don't use shared memory to communicate between processes.