what libraries or platforms should I use to build web apps that provide real-time, asynchronous data synchronization between clients?

Question

This is a less a question with a simple, practical answer and more a question to foster discussion on the real-time data exchange topic.

I'll begin with an example:

Google Wave is, at its core, a real-time asynchronous data synchronization engine. Wave supports (or plans to support) concurrent (real-time) document collaboration, disconnected (offline) document editing, conflict resolution, document history and playback with attribution, and server federation.

A core part of Wave is the Operational Transformation engine: http://www.waveprotocol.org/whitepapers/operational-transform

The OT engine manages document state. Changes between clients are merged and each client has a sane and consistent view of the document at all times; the final document is eventually consistent between all connected clients.

My question is: is this system abstract or general enough to be used as a library or generic framework upon which to build web apps that synchronize real-time, asynchronous state in each client?

Is the Wave protocol directly used by any current web applications (besides Google's client)? Would it make sense to directly use it for generic state synchronization in a web app?

What other existing libraries or frameworks would you consider using when building such a web app?

How much code in such an app might be domain-specific logic vs generic state synchronization logic? Or, put another way, how leaky might the state synchronization abstractions be?

Comments and discussion welcomed!

Answer 1

Jack Moffitt's book "Professional XMPP Programming with JavaScript and jQuery" provides the answer to exactly this question, including the Operational Transform bits.

Answer 2

HTTP was made to handle calls that were initiated from the client to the server, but this is a problem where you need Push Technology. I'm a web developer, so would use Comet.

The browser makes an Ajax-style request to the server, which is kept open until the server has new data to send to the browser, which is sent to the browser in a complete response. The browser initiates a new long polling request in order to obtain subsequent events.

You can build Comet-style Web applications using a web development framework called Lift. (It's written in a programming language called Scala that compiles down to Java bytecode, which means you can run it on a Java Application Server.)

Something interesting to watch for is HTML5, which has a feature called Web Sockets that may make Comet obsolete.

Answer 3

You can achieve this with something like PubSubHubbub and I think it's pretty simple. The way it works : both systems have to have RSS/Atom feeds which represent the data, or at least events on the data "object at url XYZ created"... etc.

Then, make sure both feeds are PubSubHubbub enabled, which means that they can notify subscribers when they update their content.

Subscribe each component to the feeds of the other components.

Implement what happens on notification : downloading the file, updating stuff in a database... you name it.

The great advantage of this is that it relies only on stuff that are "known" : HTTP, ATOM.

This blog post gives an example with social data, but that can also apply to any type of data.

Answer 4

As far as OT implementations go, Wave is really watered down and does not - and can not - live up to the promise of OT that you will read about in the literature. Being based on the Jupiter collaborative system, it is limited to a client/server network topology. Google further hobbled the Jupiter OT algorithms to limit the number of operations "in flight" by a given client to one - this really hurts interactivity between clients.

How "general" is Wave? From what I recall of the code, it seems pretty tied to the Wave data model - blips, annotations, etc. So, I expect that it would be difficult to apply it to other data models without serious modification.

Beyond that, I would be concerned about scalability and robustness of a Wave based system. For example, only a single server can handle operations on a given Wave and it's not clear how you can implement fail over support since any rollback of the server at all is pretty much catastrophic, resulting in all clients ditching their waves (including outstanding operations).

Wave is actively under development, so things will improve over time.

There aren't many alternatives to Wave. As noted in my answer to this question, I use Ceda for OT based synchronisation. There is no "leaky abstraction" problem in Ceda - it can use OT to synchronise arbitrary data structures - your application defines the schema.