Unit tests in production release code?

Question

Couple of questions:

1.) Do you unit test release code?

2.) If so, do you then leave those unit tests intact so that the tests themselves exist in the production environment?

I see the value in #1, but is it a "good practice" to create dependencies in production to, for example, the NUnit assemblies?

Give me your thoughts.

Answer 1

Yes and Yes, application behaviour can be different between release and debug builds, hence as part of the release process the release build has to pass all its unit tests.

Answer 2

1. Absolutely. If our build passes the unit test suite then it's tagged and a candidate for production
2. No. Deployments don't include tests nor the supporting libraries (e.g. unit test libraries, mocking etc.)

The above is my general rule (I'm usually deploying to non-technical users). However I do have an exception, which is a programming utility that is unit tested with ~130 test scripts. Because the test scripts double as examples, I deploy those along with the production release, and consequently they enhance the existing documentation.

Deploying tests with open-source code is definitely worthwhile. It allows people to play with, modify and submit patches, whilst being able to run the tests that passed successfully to permit the original artifact to be released.

Answer 3

1. Yes of course! Unit tests run on all build configurations.

2. Unit tests are always intact but this does not mean that shipped assemblies are dependent on anything relating to the tests. Tests are always written in a parallel assembly (in the same build environment) that then tests the production assembly. The parallel assembly is not shipped as it only contains the tests.

Answer 4

1. Yes, remember the classic "Assert with side-effects" error that must be caught as well. But this one doesn't need to be done as often as the debug build, where a complete test should be done every day.
2. Typically unit-tests are in different translation units and in a different project, so that a release build of the main project does not touch them at all. If your unit-tests are in the same translation units as the tested code though, you can use conditional compilation to exclude them from releases.

Answer 5

Depends on the project. Yes to number 1. Following the principal that everything should be checked into source code control and it should be simple to get a new developer going. Make them part of the codebase. New people can do a check out and run the tests.

Whether they are deployed to production is a different issue. I haven't worked on a project that needed them there. Rails' deployment model is (generally) simply a check-out of the whole project on a production machine, so yes they are there. Java/Maven projects have a whole build/packaging step, and generally unit tests can-- and are-- removed when building the final .war file.

Either way, you don't expect them to run. In today's environment, it doesn't really matter if they site there-- memory and disk are so cheap it's really not an issue. I have heard the argument that you don't want the test code on the production server so that there's no risk that it will get run, but I haven't heard of a scenario when this would really happen.