How to ensure message transfer by pull-/polling from a webserver.

Question

This builds on How to send messages between Companies. If I decide that company S(upplier) should poll orders from company (B) in some simple HTTP based Way what is the best implementation.

• I assume Company B has a Webserver running and the backend database of this Webserver is durable. We should make as few a possible assumptions about the storage processes at S and if they are able to keep state (e.g. a list of already transmitted GUIDs)
• The Internet connection between B and S is unreliable.
• We have to reach eventual consistency meaning at one point in time all orders between B and S should be transferred.

What is the best practice to implement such a system?

Answer 1

One approach to this kind of problem is to use some kind of queueing product, as an IBM person I immediately consider MQ. However as I'm not actually MQ person myself, like you I would probably be happy with service based approach you are taking.

There are two possible approaches that come to mind. One is to use the WS Reliable Messaging, which pushes the reliability problem down into the Web service infrastructure. The other is to hand-crank your own reliable protocol on top of simple, but unreliable, services.

I've not got serious practical experience of implementing a system with WS Reliable Messaging, I do believe that it can be made to work, but it does require some degree of control over the participants - as it's a comparatively recent standard we can't guarantee that any given IT shop will have an implementation to hand, and interoperability between vendors may be an issue. The more control I have over the SW stacks at each end the more inclined I would be to use WS Reliable Messaging. [I should mention WS Atomic Transaction too, which also can be used to build realiable services, the same inter-op concerns apply.]

So what about roll-your-own? The key here is to make all services idempotent. As we don't have transactional guarantees that span the two systems we must assume that any given service call may fail with unknown outcome.

I'm going to assume that B wants to have confirmation that S has taken an order, therefore we need to update information at both B and S when an order is transferred.

B must offer services such as these:

 Give me the next order(s)

 I have stored {orders ...}

So how do we define "next". The simplest case works nicely if the volumes we are dealing with can allow us to have a single "thread" of transfer. Then B is ticking off the sent orders one at a time, and the orders have a monotonically increasing ID. We can then simplify to:

 I have stored order <65,004> please give me the next

note that this is an idempotent request: it can safely be repeated many times. Also note that S must anticipate the possibility of getting the same order twice, and check for duplicates.