JUnit Testing for Automated Testing Scripts?

Question

I recently joined this organisation that I am currently working at that has asked me to re factor, extend and maintain an existing Automated Testing Framework written in java, which uses a keyword driven framework and RFT. I have been a developer all my life. By habit I write unit tests to test for behavior before writing source code. This framework does not have one unit test. My first instinct was "where are the unit tests?" I know I can write unit tests for the testing framework classes. During a discussion here it was brought up that writing unit tests for testing frameworks or scripts may be a waste of time. I diplomatically disagreed.

Question 1: Could my instinct be wrong? Do you have any suggestions which can be helpful to fight my case.

Question 2: And could this get recursive? Writing tests for tests and tests and so on. Is there rule for when to stop writing unit tests? Is there a concept of testing the tester recursion?

Again I am all for unit tests but never been in this kind of situation before. I could not find much on this topic from my research.

EDIT

Thank you all for your interesting responses! The unit tests will definitely be written without doubt! Highest priority will be given to our self written framework classes and methods that are used most often and would have high ROI and high Penalty for failure. The plan is to gradually and incrementally achieve a high level of code coverage for the whole project (java)

Answer 1

If your organization wrote this testing framework, then you should unit test it. If you use an existing testing framework (JUnit for example), then I wouldn't unit test that. Leave that testing to the creator of the testing framework.

Bottom line: you wrote it, you test it.

Answer 2

To test something, you need a reference, something to compare the results to. For the framework or the script, the production code can be used as reference - unless the versions are not compatible: if all tests pass with framework N and with framework N+1, then there is no [visible] regression, will all due restrictions (provided enough coverage...). That's where writing unit tests for testing frameworks or scripts may be considered a waste of time.

The existing frameworks probably works in most cases, so spending time to put it under unit tests can be a waste. As with any piece of software, writing unit tests when you add new features, or when you rework some parts of the code will be helpful.

I usually don't write unit tests for my test programs, or only for specific parts where automated tests are valuable. I grow them along with the production code, using each one as a scaffolding for the other.

Answer 3

There was a very good interview with Kent Beck on Software Engineering radio, where he explained his philosophy on when to write tests, when to do TDD etc.

He said that when he writes a exploratory bit of code, code that won't be shared or code that won't last, a spike solution, he doesn't write tests. Pretty much the rest of the time he writes tests.

To answer your question, ask yourself a question. Would it help you to refactor, extend, maintain this framework if you had tests? If yes, write the tests.

1) Usually, good practice recommends that you write tests before refactoring (to make sure that behaviour doesn't change), or before doing new code (standard TDD).

2) Yes this can get recursive, but you only have so much time in the day, so you have to think about the effort you are expending for the extra value you're bringing to the project.

Writing unit tests can help you understand the existing code better as well. Personally, I would be writing tests.

Answer 4

Another thing to consider is that the automated tests themselves are testing the automated testing framework. That is, if you break the framework, the tests themselves ought to fail. In light of that, it may not make sense to invest in writing automated unit tests of the framework itself.

Answer 5

Question 1: Could my instinct be wrong? Do you have any suggestions which can be helpful to fight my case.

Your instincts are not wrong. If your testing framework has a bug it might miss errors by skipping tests, for example.

Question 2: And could this get recursive? Writing tests for tests and tests and so on. Is there rule for when to stop writing unit tests? Is there a concept of testing the tester recursion?

No. Because test cases are supposed to be so simple that bugs can't survive scrutiny. Every test should be trivial or the class being tested needs refactoring. Obviously sometimes this is easier said than done.

I'd examine the test framework and add tests wherever a failure would really hurt.