Is mutation testing useful in practice?

Question

It exists the technology of mutation testing. It checks, if the tests are running even if you change the code. If not all is OK, if the tests are running they don't cover all eventualities. There is some theoretical work about it, but I'm interested in the question, if it is useful in practice? Do you have any examples of real life applications of mutation testing? Does it work better than simple test-coverage-tools? Or is it useless?

What are the advantages/disadvantages of mutation testing in the real world?

Answer 1

I looked at mutation test some time ago as a method for checking the efficacy of my automated regession testing scripts. Basically, a number of these scripts had missing checkpoints, so while they were exercising the application being tested correctly, they weren't verifying the results against the baseline data. I found that a far simpler method than changing the code was to write another application to introduce modifications to a copy of the baseline, and re-run the tests against the modified baseline. In this scenario, any test that passed was either faulty or incomplete.

This is not genuine mutation testing, but a method that uses a similar paradigm to test the efficacy of test scripts. It is simple enough to implement, and IMO does a good job.

Answer 2

I recently did some investigations on mutation testing. Results are here:

http://abeletsky.blogspot.com/2010/07/using-of-mutation-testing-in-real.html

In short: mutation testing could give some information about quality of source code and tests, but it is something straighforward to use.