Application Architecture - How would you design this process

Answer 1

I would not go through each record in the scenario you describe. Web services are fine but you could as easily use them for bulk updates.

From the top of my head, something like this would work a little better:

1. Get a 'diff' containing all the changes in the master database. This could be done before the synchronisation starts and only has to be done once per day.
2. Send a full list of contacts from the client to the server. With a list that big it does not really matter how it is send but if you don't like SOAP web services think about JSON. I would go with web-services for convenience. Again, the list of contacts could be prepared in advance.
3. Create a list of records which need to be update by comparing the 'diff'with the list from the client.
4. Get all the updated details in one go (if you use MS SQL you could do that with XML, other SQL services offer different paths)
5. Send the details to the client

The same mechanism could be used for updating one record only, you just need to use different list (containing 1 ID) in the p.2

Answer 2

As you're relatively constrained by the third party, you might want to do one of the following:

1. Have the third party create a web service which returns a list of all id's which have details changed in the last day.

2. You then iterate through this reduced list using a web service (which hopefully is much smaller then the entire list), and update your contacts as appropriate.

or

1. Have the third party create a web service which contains an xml document containing a dump of the contacts details in their system which have changed in a certain period. You would then process this document, updating your details.

Answer 3

You can ask for a file (XML) containing all the ID details from the central DB and you can get it converted in to some object mappings. Then having object list at your end you can compare and update accordingly.

Answer 4

Go with bulk transfers wherever possible.

Opening a TCP/IP connection is a VERY slow process. Transferring 20K contact ID's (even if the ID is huge) in a single WS request is a lot faster than 20K web service requests.

You don't want to wait for them to reply. You have two kinds of potential workflows.

1. Busy Waiting. You send a batch. You poll every few minutes until the batch is done. This is rather icky, but totally one-sided. You do all the work, they just respond with "not done yet" or "done". Then you request your batch results.

2. Notification. You send a batch including some address at which you can be notified. It might be an email address, or it might be a URL (IP address, port and path) at which you want notification.

   • If you elect to use email (or similar) notification you can then do a proper WS request to retrieve the batch.

   • If you elect to use WS notification, you have a small web service that either gets a simple notification, and does a WS request to get the result, or they send you the entire result.

3. If you want to be rude, open several hundred threads and make several hundred concurrent requests. This will take less elapsed time, but will swamp their server.

Answer 5

how about messaging? On the central database side, a messaging queue system can be created for storing db records update events, and there may be a topic for each client databases so that client can only subscribe to those interesting events. The queue will be responsible for notifying client of records update events.