Tagging unit-tests

Question

I'm working on a PHP project with solid unit-tests coverage.

I've noticed, that last time, I'm making very tricky manipulations with unit-tests Command-Line Test Runner' --filter command.

Here is this command's explanation from official documentation:

--filter

Only runs tests whose name matches the given pattern. The pattern can be either the name of a single test or a regular expression that matches multiple test names.

I ofter use it because sometimes it becomes very useful to run just a single test suite or test case from the whole test base.

I'm wondering if this is good practice or not?

I have heard that sometimes it is good practice to to run the whole test suite on your Continuous Integration machine, if you know for sure that you have modified only one component and 100% percent confident, that it won't fail other component's unit-tests. What do you think about it?

Some time ago I thought that we shouldn't care so much about time require to run the whole suite of all unit-tests, but when you have very complicated business logic and unit-tests - this can take significant time.

I understand, that "real" unit-tests shouldn't interact with DB, use mock/stubs objects, I agree with that. But sometimes, it is much easier(cheaper) to use DB fixtures for the tests.

Please give me some advice, how this problem can be solved?

Answer 1

Good unit tests should:

• Have clear methods names and variable names to act as documentation
• Run fast. This will also be possible for test with complicated business logic. Test should run in an avarage time of something around 0.1 second.
• Test exactly one thing in one test method
• Not integrate with external resources like the filesystem, email, databases, webservices, and everything else. You can create seperate database integration tests to test your database ineraction. These test will slower then your unit test most of the time. I put my integration tests in a seperate project and I run them only when I am working on the integration code. I also run them on all builds on the CI server.
• Be completely isolated from each other. When you have tests depending on each other, you cannot see what your problem is from reading which tests are failed. You might have to debug to find the problem. Isolated tests will save you a lot of time.

Personally, I don't use category names in my tests. I use 2 test projects per application. One for the unit test and one for the integration tests and slower tests.

Reaction on:

"But sometimes, it is much easier(cheaper) to use DB fixtures for the tests."

When your code is written well, it will be easier to mock. I don't know about mocking frameworks in Php, but I use them in other languages to save me a lot of time. Writing test first and code later might help you to design your code to be testable easier.

Personally I learned to test better by

• reading blogs about it
• reading books about it
• reading tested code written by others
• writing a lot of tests of course. It took me a few thousends of tests to become good at it.

Answer 2

I ofter use it because sometimes it becomes very useful to run just a single test suite or test case from the whole test base.

I'm wondering if this is good practice or not?

Sure, as long as you run the full set of unit-tests occasionally (via a CI server sounds perfect)

Running the "interesting" tests regularly is better than running all the tests rarely..

Answer 3

I'd address the issue by having a subset of tests ("smoke tests") that take 1 minute or less that must be run before committing, then run the full set of tests from your CI server.

If your full set of tests takes > 15 minutes then I'd look to divide them and run them in parallel.

Then you can use the --filter to run the tests you're most interested in first, then the smoke tests prior to commit, and have the rest run from the CI server.