Should my unit tests be touching an API directly when testing a wrapper for that API?

Question

I have some written a number of unit tests that test a wrapper around a FTP server API.

Both the unit tests and the FTP server are on the same machine.

The wrapper API gets deployed to our platform and are used in both remoting and web service scenarios. The wrapper API essentially takes XML messages to perform tasks such as adding/deleting/updating users, changing passwords, modifying permissions...that kinda thing.

In a unit test, say to add a user to a virtual domain, I create the XML message to send to the API. The API does it's work and returns a response with status information about whether the operation was successful or failed (error codes, validation failures etc).

To verify whether the API wrapper code really did do the right thing (if the response indicated success), I invoke the FTP server's COM API and query its store directly to see if, for example when creating a user account, the user account really did get created.

Does this smell bad?

Update 1: @Jeremy/Nick: The wrapper is the focus of the testing, the FTP server and its COM API are 3rd party products, presumably well tested and stable. The wrapper API has to parse the XML message and then invoke the FTP server's API. How would I verify, and this may be a silly case, that a particular property of the user account is set correctly by the wrapper. For example setting the wrong property or attribute of an FTP account due to a typo in the wrapper code. A good example being setting the upload and download speed limits, these may get transposed in the wrapper code.

Update 2: thanks all for the answers. To the folks who suggested using mocks, it had crossed my mind, but the light hasn't switched on there yet and I'm still struggling to get my head round how I would get my wrapper to work with a mock of the FTP server. Where would the mocks reside and do I pass an instance of said mocks to the wrapper API to use instead of calling the COM API? I'm aware of mocking but struggling to get my head round it, mostly because I find most of the examples and tutorials are so abstract and (I'm ashamed to say) verging on the incomprehensible.

Cheers Kev

Answer 1

What are you testing the wrapper or the API. The API should work as is, so you don't need to test it I would think. Focus your testing efforts on the wrapper and pretend like the API doesn't exist, when I write a class that does file access I don't unit test the build in streamreader...I focus on my code.

Answer 2

I would say your API should be treated just like a database or a network connection when testing. Don't test it, it isn't under your control.

Answer 3

I agree with Nick and Jeremy about not touching the API. I would look at mocking the API.

http://en.wikipedia.org/wiki/Mock_object

If it's .NET you can use:
Moq: http://code.google.com/p/moq/

And a bunch of other mocking libraries.

Answer 4

It doesn't sound like you're asking "Should I test the API?" — you're asking "Should I use the API to verify whether my wrapper is doing the right thing?"

I say yes. Your unit tests should assert that your wrapper passes along the information reported by the API. In the example you give, for instance, I don't know how you would avoid touching the API. So I don't think it smells bad.

Answer 5

You seem to be mixing unit & component testing concerns.

• If you're unit-testing your wrapper, you should use a mock FTP server and don't involve the actual server. The plus side is, you can usually achieve 100% automation like this.
• If you're component-testing the whole thing (the wrapper + FTP server working together), try to verify your results at the same level as your tests i.e. by means of your wrapper API. For example, if you issue a command to upload a file, next, issue a command to delete/download that file to make sure that the file was uploaded correctly. For more complex operations where it's not trivial to test the outcome, then consider reverting to the COM API "backdoor" you mentioned or perhaps involve some manual verification (do all of your tests need to be automated?).

Answer 6

To verify whether the API wrapper code really did do the right thing (if the response indicated success), I invoke the FTP server's COM API

Stop right there. You should be mocking the FTP server and the wrapper should operate against the mock.

If your test runs both the wrapper and the FTP server, you are not Unit Testing.

Answer 7

The only time I can think of when it might make sense to dip into the lower level API to verify results if if the higher-level API is write-only. For example, if you can create a user using the high-level API, then there should be a high-level API to get the user accounts, too. Use that.

Other folks have suggested mocking the lower-level API. That's good, if you can do it. If the lower-level component is mocked, checking the mocks to make sure the right state is set should be okay.

Answer 8

To test your wrapper with a mock object, you can do the following:

• Write a COM object that has the same interface as the FTP server's COM API. This will be your mock object. You should be able to interchange the real FTP server and your mock object by passing the interface pointer of either to your wrapper by means of dependency injection.
• Your mock object should implement hard-coded behaviour based on the methods called on its interface (which mimics FTP server API) and also based on the argument values used:
• For example, if you have an UploadFile method you can blindly return a success result and perhaps store the file name that was passed in in an array of strings.
• You could simulate an upload error when you encounter a file name with "error" in it.
• You could simulate latency/timeout when you encounter a file name with "slow" in it.
• Later on, the DownloadFile method could check the internal string array to see if a file with that name was already "uploaded".

The pseudo-code for some test cases would be:

//RealServer theRealServer;
//FtpServerIntf ftpServerIntf = theRealServer.getInterface();

// Let's test with our mock instead
MockServer myMockServer;
FtpServerIntf ftpServerIntf = myMockServer.getInterface();

FtpWrapper myWrapper(ftpServerIntf);

FtpResponse resp = myWrapper.uploadFile("Testing123");
assertEquals(FtpResponse::OK, resp);

resp = myWrapper.downloadFile("Testing123");
assertEquals(FtpResponse::OK, resp);

resp = myWrapper.downloadFile("Testing456");
assertEquals(FtpResponse::NOT_FOUND, resp);

resp = myWrapper.downloadFile("SimulateError");
assertEquals(FtpResponse::ERROR, resp);

I hope this helps...