How complex should smoke tests be?

Question

So we've been running a daily build on our current project for a lot of months at this point. The smoke tests that goes along with that daily build isn't very complex, though - we run a few nUnit tests on our main class library (which, admittedly, doesn't offer great code coverage), and we make sure that things compile and build. The application in question is an ASP.NET site which consumes some business objects (which include LINQ-to-SQL).

Are there more complex smoke tests that we should be running, particularly on the ASP.NET sites? How would we develop a smoke test for an ASP.NET site, for that matter?

Answer 1

As well as unit testing, it can be good to launch the site to a staging server with some example data. As close to live-like as possible. Then use a HTTP traffic generating script to simulate user traffic and sessions. You can monitor debug logging, exceptions and other testing code on the back-end. You can also take performance measurements here.

Much like a more intense, iterative version of playing with it in the browser yourself.

You can do this by defining (or through inspection) your public resources and their inputs. The scripts can then try and cause validation problems, odd permutations of site flow and other things that test the entire context of the site in a live setting.

If testing is not complete ... from unit testing through to "does it play nice with real data and traffic" then you are ultimately going to be running around like a headless chicken fixing bugs later.

Answer 2

You should not be doing smoke tests. Are you aware of the etymology of that term? A "smoke test", in electronics, is when you turn on the power and see if any smoke comes out.

You should be doing more comprehensive unit tests; enough to give you good code coverage. This is what you should do on every build. You should also try to do a deployment, and run some "installation verification tests".

Answer 3

Smoke tests, by nature, should be superficial: Does it compile? Deploy? Does the welcome page load? Maybe load a test page which does a query against the database to see that this connection works, too. That's it.