Testing clients for public web services

Question

What are approaches to test custom clients for public web services?

Today there are many online services which provide an API. There is a boom of little apps using those APIs. Examples: desktop/mobile clients for social networks and blogging platforms, document storage and processing centers, cloud databases, real-time data streams, GIS data etc.

The problem is that often the non-trivial part in such applications is communicating with the online service (handling errors, encoding/decoding data, dealing with quotas, adjusting to API updates etc.), but developers of the client do not control the service. So one cannot see directly what effects of the tests are, and one cannot always rollback the state of the service back to the original.

• how do you design your client tests to be reproducible?
• what behaviors do you test?
• how do you test destructive or heavy-load behaviors? (against a public service)
• do you run such tests automatically (e.g. as a pre-commit hook)?
• how do you test against extraordinary situations (from service is down to exceeding quota, to inconsistent state, to sudden change of service behavior)?

Answer 1

Be very clear what you are testing. Are you testing that your code does what it's supposed to when it receives responses from the service? Both normal and extraordinary? Then mock the service, so that you can readily exercise those paths.

Yes I would design reprodicible tests and run them under some framework that lets me run them automatically, ideally as part of a build/commit too.

But what about testing the service itself. Some tests just emerge from validating your solution. For example heavy load. Well, although it's important not to be anti-social and it's not reasonable to saturate a public service, if there is a publised SLA, then I think it's reasonable to test up to that. So if your app is expected to emit n requests a second, then surely we should test at least this. Testing our overall solution up to its required throughput.

To destruction? Maybe too anti social. However I do think that sending both valid and invalid requests and checking that expected responses occur may be valid and valuable if only as a sanity check of the service you're using. So I would have at least a regression-suite for the public service, so that I can readily validate that it is behaving as documented.