How to organize live data integrity tests and code unit tests?

Question

I have several files with code testing code (which uses a "unittest" class).

Later I found it would be nice to test database integrity also. I put this into a separate directory tree. (Things like the keys have correct format, parent and child nodes are pointing correctly and such. Edit: this is a nosql project, where I can not rely on database level checks liek referential integrity and such.)

I use the same unittest class for the integrity tests.

Now I wonder if it makes really sense to keep this separate. To test the integrity of data I often duplicate parts of code that I use to test the code that handles the data.

But it is not the same. The code tests use test databases (that get deleted after each test) and the integrity tests connect to the live data and analyze it. The integrity tests I want to call from cron and send an alarm if something happens in the live database.

How would you handle that? Are there standards for such a setup? What is your experience?

My tendency is to put everything in the same file, which would result in the code tests also being executed by the cron on the production environment.

Edit: What also drives me is to try to keep the project simple and not to have too many files touched by a single task or workflow. Without all the testing i already have a class file, a subclass, a related class, some library (helper) files and the main code. Testing adds one file. It helps me keep my attention focussed while coding, it is less stressing and I believe I make less errors, and I can faster remember and find a certain code part with less files affected. Only one testing file per workflow would help here. If I keep it seperate there are 2 files (data integrity testing and code testing) and maybe 3 (a common library for both). Abstraction would add complexity.

Edit2: I am now refactoring a little bit and only moving the data testing files to the same directory tree where also the code testing lives, but keeping different files with the name indicating "integrity" or "testing". I will not (yet) merge the files, because 2 people recommended against it, and I believe in their experience and advice for now. I will live with code duplication for the moment.

Edit3: I forgot to mention that the selection of the tests per run is not determined by the tree structure in this case. The tests are enumerated in a master file, so I have 2 master files "integrity" and "code testing" at the present, and the test can live in the same directury structure.

Maybe more people will answer. Thank you so far for the valuable input, which is already helping me develop the final structure!

Edit4: I did more refactoring now. It seems I should keep 2 files, but with slightly modified purpose. One targeted for scheduled monitoring on the production server. And another one for development. But in both files can be integrity tests or code tests. And in both files operations can be performed on test databases (that are erased after the test) and on the permanent database (each one has a permanent database, production server and develpment server). And one important thing: I find myself moving lots of common code from the testing files to the class files. So the classes get also abilities that are for testing only. I like this so far, feels clean. I am not (yet) creating a library for testing that is shared between the 2 testing frontends, this code has gone to the class file of the obejct that is being teted for now.

Please note that my comments below are signed with "user89021" but it's me, karlthorwald. I can't do anything about it.

Answer 1

Your approach of separating them is good.

Your concern about code duplication is 100% valid.

The solution is fairly straightfowrard - abstract away common functionality between the tests - e.g. "RunTest", "AnalyzeResult", "ConnectToDB" - into a common library (you did not specify which language but I assume it has a concept of a library) which can be passed configration details such as which database to connect to.

Then use that library independently from the unit test driver and integrity test driver - which, if you are skilled/lucky enough, might have very little code of its own other than configuration (e.g. which database to connect to, how to report results, and which tests to run).

Similarly, if needed, common inputs/datasets can be placed in common directory

Answer 2

You should separate the database related tests from the "pure" unit tests.
The cost of having two different assemblies is very low considering the benefits - you have one suite of fast, no environment set required tests that you can run on any machine and a slower suite that tests the database integrity that can run only on specific places (e.g. build server).

Another benefit is that you can have two build processes (quick and nightly) that runs different tests suites.

To avoid duplicating code you can create another assembly with the common methods/actions that both test suites needs. Don't worry too much about duplicatimng the actual tests because you're testing different things (either logic or database) so sooner or later your tests will become quite different depending on what you're trying to test.

Answer 3

One more answer. You have two types of tests. What I would like to do is address the integrity tests. What you may want to do is include the integrity tests as a function of the production code. IOW, have the integrity as part of the system.

You already mentioned that duplication is an issue and that you are refactoring to remove the duplication. The refactored code of course has development tests?

System Monitoring can be production code. So what ever code you write becomes part of the system.

The nice thing about this is that you evolve your code through your development tests.