How does the garbage collector work with unit tests?

Question

Recently, I asked (and answered) a question on StackOverflow about why a unit test would work when run by itself and then fail sporadically when run with the whole batch of unit tests. See here: http://stackoverflow.com/questions/3072986/sql-server-and-transactionscope-with-msdtc-sporadically-cant-get-connection

Unit tests passing when running one at a time and then failing when run together is a classic sign that something is seriously wrong with the code.

I discovered that there is a bit of a resource-leak. Because of a subtle error causing connections to a SQL server to not be released, I was running out of connections and tests were failing. AFAIK, this works almost exactly like a memory leak; connections are allocated from a connection pool and never freed just as memory can be allocated and then not freed.

However, this does leave me with a puzzling question? What is the difference between running my tests one at a time and running them as a suite? If the tests pass when run one at a time and then fail when run together, then there must be some sort of clean-up happening between test runs that happens only when the tests are run one at a time.

I conjecture that this could have something to do with what the .net garbage collector does or doesn't do between the tests. In one case, connections are freed between tests; in another case, they aren't.

How can I explain this?

Update: To those of you asking about the specifics of the code, it's rather simple. I declare a new TransactionScope object in my Setup mehtod and dispose it in my Teardown method. However, The problem test was a data-driven test with 100 test cases; the code under test populated a SqlDataReader object from a select statement using the SqlHelper class and then didn't call the close method on the SqlDataReader. Because I used the SqlHelper class to get the SqlDataReader, I expected that the connections were handled for me. Not so!

But to clarify, I'm not asking about my specific situation. What I want to know is: generally, how are resources freed between tests? I would imagine this would be some application of the garbage collector. I wonder if the garbage collector could still be cleaning up a previous test as the next test runs (race condition?)

Update: What I know about garbage collection with Unit Tests. Following my own curiosity, I pulled out the unit tests that were failing because a connection was left open by the SqlDataReader object. I tried adding System.GC.Collect() to the end of each test. This successfully freed the connections, but does impose an ~50% performance penalty.

Answer 1

Garbage collection is a periodic background task. In specific, there's a thread that does nothing but finalize objects that have already been marked as dead. By running one test at a time, you're giving that thread a chance finalize objects so as to close the connections.

Answer 2

That sounds feasible, yes. It wouldn't be at all surprising for the unit test framework to request that the garbage collector runs between tests.

Alternatively, the different pattern of execution may just naturally trigger garbage collection when they're run one after another. The trouble with analyzing this sort of thing is it's all very dynamic - and will vary from test run to test run.

Don't forget it probably didn't have to free all the connections between tests - just enough to keep them running...

The garbage collector itself is unlikely to behave any differently in unit tests, unless the test runner process is configured in a particular way. On the other hand, whether you run the tests in the debugger or not will affect how eager the garbage collector is, etc.

Answer 3

Usually each test run is performed in a separate appdomain for several reasons. Now when the appdomain is unloaded it will release the resources associated with it, so that the open connections are closed and therefore preventing the "leak" to manifest itself.

See also Cbrumme's blog on this topic.

Answer 4

Unit tests passing when running one at a time and then failing when run together is a classic sign that something is seriously wrong with the code.

I think there is something seriously wrong with the way you have written your unit tests. Each test should run independently of other tests. One way to do it is ensure you have a setup and teardown methods([SetUp][TearDown]) which create and cleanup the environment necessary for a test to run.

In your Setup method you create your connection, in your teardown method you dispose it. Now before running each test, your Setup method will be called and after each test your teardown method will be called and this will ensure that you don't leak any resources.

Answer 5

Wow, multiple issues here!

First, you want your unit test series to be FAST. Don't hit the database to test business logic, etc.

Second, if your production code is leaking resources (?) that is your main problem. Don't fix that problem by changing how you setup/teardown your test code. Now, if your test code is allocating system resorces but not disposing correctly, you need to fix that the right way and not by trying to control when the garbage collector runs. You should not have to worry about that.

Third, you really should not have to create a TransactionScope in your unit tests. This makes no sense to me. There is something wrong with the coding style you are using in your test code. Unit tests are not just any automated tests, such as integration tests or system tests. Unit tests are small and focused tests that test the behavior of a SMALL piece of production code in ISOLATION, that is independently from all the other production code.

Now, a tip about the leaking resources. A good programming practice is to use the using-statement when creating a disposable object to guarantee those resources are properly disposed.

using (SqlDataReader reader = ...)
{
   ...
}