Dependencies on external (web) services

Question

I'm currently involved in a project where we are developing a large website that relies heavily on an external service (for some functionality) developed by another company. The external service occasionally breaks and doesn't provide us with the data that we need. This is a major problem for us since the requirements on "our" website are very high.

How should we handle this? We are reluctant to cache data from the external site to use as a "backup" since we might then display data that is outdated or wrong. We also feel that we should not try to "patch" problems in an external system by storing local copies of the external data since that could lead to synchronization problem where the local data is out of date or wrong.

Does anyone have any similar experiences? Any ideas how we solve this (or at least mitigate the problem)?

Answer 1

Caching would be my first choice.

It depends on the site your developing, but could you not cache all results and inform users that this is what you have done - for example "this was accurate on 2009-10-08 - click here to refresh"

UPDATE: Without knowing more about what sort of data you are getting from the web service and what the audience of your website is its hard to know what to suggest as the solution will depend a lot on those factors. You need to think about your customers to decide whether or not you can show them potentially out of date data or not.

If the requests to the web service tend to be similar then one thing to consider is the possibility that caching will also help you with performance and scaling as well as helping with resilience.

Answer 2

You could use caching as a backup mechanism for when the external resource is not available. In this way you are using live data for as long as the service is available and when it goes down is when you dig into your cache. It would be important to somehow mark the data in such a way that the consumer of that data knows how fresh it is.

You know what your needs and requirements are and based on what I can read the freshness of the data is critical but so it the availability of your service. Based on this understanding if you can cache data I would definitely do that but only use it as a backup when needed.