I already solved this but thought I'd ask anyway...
I have a large application with a reasonably comprehensive test suite and unit test system. I was working on it and made a change that shouldn't have broken anything and suddenly dozens of test are failing. What should I do?
I'm looking for a list of things that help isolate bugs. Tips & tricks. That kind of thing.
I'll assume that test are atomic as in they can't be split or I can find the exact point in the test that an error is detected.
Quickly scan the failures. Find a case that you have some guess as to what's wrong. Debug that one.
The theory being that there is really only one or two bug (if you changed enough that that is a questionable assumption, good luck) and fixing one test cases will fix many others as well so you might as well fix the bug with an approach that is easy.
Compare the new version of the code with the previous (tested) version and find your bug?
I'm not sure that answers your question though. What am I missing?
Having a good test suite goes a long way to debugging changes. Pat yourself or your team or your predecessor for doing things the right way. Those of us without test suites don't want to hear about it. ;-)
One interesting experiment is to test out the original branch. Check out in a different directory and then run you test suite. Make sure that works properly. Then, I find the best thing is to compare a directory that's working fine with the one that's broken and just do binary search on a single representative test -
I often run one failing TestCase in isolation. I'll also turn on debug logging and take a close look at just one test.
If necessary, I've even created very focused test suites for this kind of work.
I would suggest that you take the simplest test that it is failing on and split that test up into two smaller tests. If one fails and the other succeeds then repeat. Essentially doing a binary split on the test until you get down to the lowest level and find the bug.