Webservice test isolation - but when to verify the webservice itself?

Question

I am isolating my webservice-related tests from the actual webservices with Stubs.

How do you/should i incorporate tests to ensure that my crafted responses match the actual webservice ones (i don't have control over it)?

I don't want to know how to do it, but when and where?

Should i create a testsuite-testsuite for testdata testing?...

Answer 1

If you can, install the service in a small, completely controlled environment. Drawback: You must find a way to be notified when a new version is rolled out.

If that's not possible, write a test that calls the real service and checks for vital points (do I get a response? Are all parts there and where I expect them? Can I parse the result?)

Avoid things like checking timestamps, result size, etc., that is things that can and do change all the time.

Answer 2

I would use something like this excellent tool

Storm

Answer 3

You can test the possible failures using EasyMock as follows:

public void testDisplayProductsWhenWebServiceThrowsRemoteLookupException() {
     ...
     EasyMock.expect(mockWebService.getProducts(category)).andThrow(new RemoteLookupException());
     ...
     someServiceOrController.someMethodThatUsesMockWebService(...);
}

Repeat for all possible failure scenarios. The other solution is to implement a dummy SEI yourself. Using JAX-WS, you can trivially annotate a java class that generates an interface consistent with the client you consume. All of the methods can just return dummy data. You can then deploy the services on your own server and point your test environment at the dummy location.

Perhaps more importantly than any of the crap I've said so far, you can take the advice of the authors of The Pragmatic Programmer and program with assertions. That is, given that you must inevitably make certain assumptions about the web service you consume given that you have no control over it's implementation, you can add code such as:

if(resultOfWebService == null || resultOfWebService.getId() == null)
    throw new AssertionError("WebService violated contract by doing xyz: result => " + resultOfWebServivce);

That way, if your assumptions don't hold, you'll at least find out about it instead of potentially silently fail!

You can also turn on schema validations and protocol validations to ensure that the service is operating according to spec.