How do you make a unit test when the results vary?

Question

I am building an application that queries a web service. The data in the database varies and changes over time. How do I build a unit test for this type of application?

The web service sends back xml or a no search results html page. I cannot really change the web service. My application basically queries the web service using HTTPURLConnection and gets the response as a String.

Hope that helps with more detail.

Answer 1

Abstract out the web service using a proxy that you can mock out. Have your mock web service return various values representing normal data and corner cases. Also simulate getting exceptions from the web service. Make sure you code works under these conditions and you can be reasonably certain that it will work with any values the web service supplies.

Look at jMock for Java mocking.

Answer 2

It sounds like your testing at too high a level. Consider mocking the web service interface and writing other unit tests on the data layer that access the database. Some more detail here might make this question easier to answer, for example the situation you're trying to test.

I would normally expect the results of a unit test not to change, or at least to be within a range that you're expecting

Answer 3

Your question is a little open-ended but there are definitely some testable options just using the information above:

1. You could test whether the query works at all. Assert that you should get back a non-empty / non-null result set.
2. You could test whether the query results is a valid result set. Assert that the results should pass your validation code (so at this point, you know that the data is non-null, not non-sensical and possibly useful).
3. If you know anything about the data schema / data description, you could assert that the fields are sensible in relation to each other. For example, if you get a result with a helicopter, it shouldn't be associated with an altitude of negative 100 meters....
4. If you know anything about the probabilistic distribution of the data, you should be able to collect a set of data and assert that your resulting distribution is within a standard deviation of what you'd expect to see.

I'm sure that with some more information, you'll get a pile of useful suggestions.

Answer 4

Strictly speaking of unit-testing, you can only test units that have a deterministic behavior.

A test that connects to an external web server is an integration test.

The solution is to mock the HTTPURLConnection - that is, create a class in your unit tests that derives from HTTPURLConnection class and that returns an hardcoded, or a parameterizable value. EDIT: notice this can be done maunally, without any mocking framework.

The class that queries the web server shall not instanciate the HTTPURLConnection, but receive it via a parameter. In the unit tests, you create the HTTPURLConnectionMock, and passes it to the class that interrogates the web server which will use it as it is using a real HTTPURLConnection. In the production code, you create a real HTTPURLConnection and pass it to the class.

You can also make your HTTPURLConnectionMock able to throw an IOException, to test error conditions. Just have a method to tell it not to return the result but an exception at next request.

Answer 5

A problem I've run into is with convoluted (meaning "crappy") datamodels, where you can't ever be sure that problems are due to code errors or data errors.

A symptom of this is when your application works great, passes all tests, etc. with mocked data or a fresh dataset, but breaks horribly when you run your application on real data.