Can a test class become a "God object"?

Question

I'm working on a backend for an open source Python ORM. The library includes a set of 450 test cases for each backend, all lumped into one giant test class.

To me, that sounds like a lot for one class, but I've never worked on a project that has 450 test cases (I believe this library has ~2000 test cases not including the test cases for each backend). Am I correct in feeling this is a bit on the high end (given that there's not really any magic number above which you should break something up), or is it just not as big a deal for a test class to have so many tests?

And even if that's not too many test cases, how would one go about refactoring an overly large test class? Most of my knowledge about refactoring is around making sure that tests are in place for the code that's being refactored. I've never had to deal with a situation where it's the tests themselves that need to be refactored.

EDIT: Previously, I had said that these were unit tests, which isn't quite true. These are more appropriately termed integration tests.

Answer 1

Rather than counting the number, I would consider if they are effective. That is, when making a single line modification, how many tests break? If you are wasting time fixing a dozen borked tests, then there is a problem; the tests shouldn't be going over the same things over and over again, and if they are, they need refactoring.

I probably wouldn't set out to refactor a test base from the top down, rather letting it flow organically from the test-write-refactor of test driven development. Write a test, implement you enhancement, and if >1 tests fail, refactor the tests

Answer 2

Split the test classes so that each class concentrates on specifying one kind of behaviour. As an example, I have written a TDD tutorial where there is roughly one test class for one kind of behaviour (in a tetris game: falling blocks, rotating pieces etc.).

Refactoring tests is equally important to refactoring the code, because the tests should provide good value as documentation, documenting the intent of what the system should do. The tests are the system's specification.

Like Todd Gardner mentioned in his answer, if lots of tests fail because of changing one method, then many of the tests are testing the same behaviour and the lines of what each test tests are blurry. This results in, that when tests fail, it's hard to know what exactly was broken, because lots of seemingly unrelated tests are failing simultanously. Also, when the system's behaviour needs to be changed, you need to also update the tests. But if the responsibilities of the tests are not clear, it will be hard to know how a test should be changed or when a test is outdated and should be removed. You might even need to change lots of tests, even though the change in behaviour was small.

450 tests in one class sounds like very much. What do the tests test, what are their names? Are they centered around the behaviour of the system (a good thing), or is there a 1:1 relation between them and the implementation? It they test many unrelated things, then it would be good to split the responsibilities to many test classes.

Answer 3

Yes, but a test class that's a "god object" doesn't sound like a problem to me.

Answer 4

450 tests in one class sounds like a lot, but how bad it is depends on how they are organized. If they all are truly independent of each other and the test class' members, it may not be a big deal - other than it must be hard to locate a specific test.

On the other hand, if the test class has members that are used by only some of the tests and ignored by others, it's a Test Smell called Obscure Test containing such Root Causes as General Fixture and Irrelevant Information (please note the jargon - I'll get back to this).

There are several ways of organizing tests into classes. The most common patterns are Testcase Class per Class, Testcase Class per Feature and Testcase Class per Fixture.

How you structure tests is important not only while you are writing the tests, but also afterwards for maintainability reasons. For that reason alone, I'd tend to say that it would be a worthwhile effort refactoring your tests. In TDD, the test code base is (almost) as important as the real code base, and should be treated with the same kind of respect.

There's a whole book about this subject, called xUnit Test Patterns: Refactoring Test Code that I can't recommand enough. It contains a complete pattern language that deals with unit testing and TDD, and all the pattern names I've used here originates from it.

Answer 5

A set of Framework unit tests are going to have a large amount of overlap in startup and setup functions. Changing a few methods could easily break every test in this case. Especially and ORM.

That said, tests should be grouped by functionality. Query of type X, unions, joins, DDL/schema retrieval, fetch caching, statement creation, etc...