Suggestions for BDD Scenarios for an generic API?

Question

I'm putting together BDD inspired unit tests for the API part of my application. (Yeah, I know, BDD is supposed to be about the domain and talking to the suits, but I'd rather try out BDD on something less visible first)

• Ordinary use. The developer uses the API methods with ordinary parameter values.

• Extreme use. The developer invokes the API with unusually large/small parameters. E.g. the zip() method is passed a 2 GB file.

• API abuse. The developer invokes the API with crazy parameters--what crazy programmer would pass in a date to an integer parameter, right?--parameters are forgotten, etc.

• Malicious hacking. The developer doesn't care what the API is intended to do, but instead is looking for ways to execute arbitrary code. Tests would include JavaScript, SQL just to see if we can get them to execute anywhere.

Are there any other scenarios I should consider?

Answer 1

Sure, there's always more scenarios to consider. There's an effectively limitless pool of scenarios, frankly. It's really an extremely open-ended question.

Regarding the malicious hacking scenarios, you should really only bother with obvious spots for buffer overflow and then stick to testing for confirmed security vulnerabilities, so that you don't accidentally reopen them. And anytime you do get a confirmed vulnerability, hunt down every place in the code that uses similar programming techniques and patterns, and preemptively test/fix those too. However, in a lot of cases, fuzzing will give you better results. Automated testing is an important part of dealing with security issues, but it should by no means be the primary tool in the toolbox.

Other things to consider are likely to be data specific. For example, when parsing dates, be sure to handle stuff like 2/29/2009 or 9/31/2009. If you can, try to handle 1/1/1900 and 12/31/2038 (your library may not let you).

One thing you can do is check the documentation for all the library calls you're making, figure out what exceptions are thrown under which conditions, and deliberately try to find inputs that will trigger those exceptions, and then make sure you've got tests that verify that those exceptions are either handled or, in the case of library code, documented.

Code coverage tools and code mutation can also help you to identify scenarios that weren't previously being covered.