Is there a pattern to synchronize data lists over network?

Question

I've got a server app that offers data to a number of clients over network. You can imagine the data as a huge list of strings. The data on the server may change and need to be synchronized on all clients.

I'm currently using this approach: On initial connection, a client app requests all current data (which may be a lot of bytes). Then, it subscribes to updates (addition,s changes and deletions) of any data.

This works fine unless the data are changed on the server inbetween sending the initial data list and the app subscribing to changes - in that case, the client misses a few updates and works on outdated data without knowing it.

I believe this is a very common scenario, so there should be a pattern that solves the issue. I'm using C# 4 and WCF, but the pattern should be language agnostic I believe.

Answer 1

I have no experience doing this, but...

If each 'initial connection' contains a 'timestamp', and when you subscribe, you subscribe 'since timestamp of initial connection' then server could include all changes since that timestamp, yes?

Key is to include some server-canonical notion of time that both client and server agree on.

e.g. instead of

void Connect()
Subscribe()

you have

Timestamp Connect()
Subscribe(timestamp)

Answer 2

I would just add another message that sends a full update to the client immediately after the Subscribe request.

Alternatively, the client could initially assume that the list is empty and the Server sends out messages to rebuild the full list on the Client by packaging them into a number of incremental "addition" messages.

Answer 3

Have you had a look at Microsoft's Synchronization Framework for .Net. It gives you finer control over what gets synchronized and how. WCF is fully supported. It supports incremental updates and synchronizing changes back to the server.

The API is pretty big and covers more than just Database Synchronization but there are plenty of articles out there. This should get you started:

Introduction to Sync Framework Database Synchronization

Synchronizing Databases

Answer 4

Maybe this article on Architecture Patterns for Data Synchronisation will be helpful to you.

It sounds like the Design Pattern for Event-Based Data Synchronisation is close to what you need.