I was reading the following article and the author made it quite clear that unit tests are NOT used to find bugs. I would like to know what your thoughts are on this. I do know that unit tests makes the design of your application much more robust but isn't it the fact that finding bugs through unit tests that make the application robust, besides its other advantages?
http://blog.stevensanderson.com/2009/08/24/writing-great-unit-tests-best-and-worst-practises/
They can actually prevent that new bugs are introduced.
A change in the application code that caused an existing unit test to fail may be a new bug that the author of the change introduced. In most cases when a unit test fails though, it's the test that needs to be updated (e.g. the code change was only refactoring).
You don't use Unit Tests to find bugs.
You use Unit Tests to make sure that your Unit of code implements all of its intended functionality properly.
Each Unit Test tests one specific piece of functionality in your code. If it doesn't work, then you're not done coding your component. It's not really a bug yet, it's just un-finished code.
Once all the Unit Tests pass, you know that your code is (in theory) fairly complete. What you can't test is whether or not it interacts with the rest of the code in the system. That's the job of Integration Testing.
This is somewhat of a semantic thing. Strictly speaking*, "Unit Tests" are there to demonstrate how code works, make sure its doing everything it should, and can also prevent a very narrow slice of regression errors. "Integration Tests" are where you will do most of your bug detection/prevention.
*From a TDD perspective
edit: The Sanderson article you reference is one of my all-time favorite articles on testing, it really crystallized for me a lot of the things I'd had floating around in my head about unit tests, integration tests, and what he calls "dirty hybrids".
As the author says, they "are not an effective way to find bugs or detect regressions." This is true. They can only find the most basic, superficial bugs that are not relevant to integration of the pieces of the application. However, without fixing these bugs, you're app won't run well enough to get on to fixing deeper and more interesting bugs.
So yes, unit tests do help you find bugs, just like compiler-errors also help you avoid bugs. But then you need to get onto real testing.
The main point seems to be that Test-Driven Development (TDD) isn't used to find bugs. That's hardly a controversial point of view.
You can also write unit tests as part of a Quality Assurance (QA) effort. You might find bugs in this way, for example by systematically covering all boundary cases.
Once again, the point is that there are more efficient ways to find bugs than by writing unit tests, but that doesn't mean that you can't use unit tests to find bugs - it's probably just not the best use of your time.
If the unit test doesn't help find a bug, what went wrong?
The unit test passes, ignoring the bug. The unit tests are incomplete or incorrect. Once you fix the tests, there will be will a failure because of the bug.
The unit test fails because of the bug. But somehow, the unit test doesn't reveal the root cause. That means another unit test could have spotted the root cause but didn't. See #1: the unit tests didn't spot the root cause because they were incorrect or incomplete.
Somehow the bug could not be found with a "unit" test. Really. That means that a mock object must be incorrectly modeling a real world object. So the mock object is incomplete or incorrect. See #1.
The "not used for debugging" doesn't really make a lot of sense, since the unit tests will tend to help with debugging. It does, however, make a little sense.
What's left are things that cannot be unit tested because they have external dependencies that cannot ever be mocked successfully.
Performance
Compliance with external source or target interfaces
End-user usability
These are notoriously hard to debug, and unit testing probably won't actually help with these few things that are totally external to the application software.
I would agree with the article.
You write unit tests to prove that your code does what you expect it to do. And to prove that it still does this after a refactor, or some other modification to that code. What you expect it to do may not be what a user expects it to do or another component expects it to do - hence these external agents can still raise bug reports against it.
The blogger appears to be saying controversial things just to be controversial.
You never write them for the purpose of finding a bug and then get rid of them. And indeed, most developers won't "find" a bug after writing a unit test. Why? Unit tests make you think about how your code is meant to be used. This means that you frequently start writing the test, then do an "oh shit!" as you realize your program won't handle this test case. So you fix your code, and then continue writing your test. No surprise that by the time you finish your test, it will pass. This is what the blogger meant by them being a "design time" activity.
Their other primary purpose is for regression tests. If you change the interface to A, then presumably some of the test cases for A will have to be changed, but the ones for B should still pass. If they don't, your code clearly didn't do what you thought it did. Time to check that design again. So yes, they are for bug finding as well.
I personally think the blogger in question goes too far when he states that unit tests are for designing robust components. I suppose they are, but not any more so than many other design activities, and they certainly don't guarantee a "robust component".
He also says "Unit tests, by definition, examine each unit of your code separately. But when your application is run for real, all those units have to work together, and the whole is more complex and subtle than the sum of its independently-tested parts. " and so unit tests can't find bugs that happen when the system is actually run.
I call these system tests. Why can't you have a set of independent unit tests, and then along side a set of system tests where the units interact with each other?
Especially a SaaS type setup :
- Service A: sending out "here I am type message with my current config"
- Service B: I've got a message from A and have set up a bunch of resources
- Service A: I'm going now, bye
- Service B: got message from A, cleaning up stuff
You could have tests where A and B interact according to some protocol and setup/cleanup as required. An individual unit test may not be able to properly test this, but these very similar system tests could.
Obviously this blogger is just stating the general case (for an application as a whole). Unit tests can and should to used to find bugs within the unit being tested
As he states:
But they [the unit tests] don’t contain any knowledge or assumptions about other parts of your codebase, so changes to other parts of your codebase don’t make them start failing
i.e. they could test for bugs within the area of logic they test, but not for the 'bigger picture'