How to make an unit test always pass?

Question

Let's assume someone has to write a solution to a problem and I have to test his solution with some tests. Is it possible (maybe with reflections or something) his program to pass all my tests, but to have nothing in common with the real solution to the problem?

Answer 1

Sure. Even without the source code to the tests, they could use a byte-code manipulator like ASM or a decompiler to correlate the inputs and the desired response.

Answer 2

JUnit 4 has @Ignore annotation, which is used in this way:

@Ignore
@Test
public void testSmth() {
   ...
}

But I'm trying to imaging the situation you've described. And the only thing I can thing about is something like ACM problems verifier (like PC^2).

And if it's the case, you'll call the code for testing and you'll have full control on the flow. So users wouln't have possibility to cheat in this way.

Answer 3

Absolutely.

If the students discover what your tests are looking for, they can write code which produces that result, without solving the homework problem.

And reflection? Yes, if you keep your expected results in variables. I'm not sure if it's possible to read inline constants. But regardless, run their code in a sandbox with a security manager. You want to do that anyway: what if their code is malicious, or has unintended consequences?

Answer 4

General Case

In the general case, no. Since they do not even know you implemented the test correctly, nothing they do - not even correctly implementing the homework - can guarantee passing the test.

Special Cases

If the students can run your test, they could use a learning algorithm.

If the students have access to the source code of your test, they can look for tricks like using reflection to get expected results stored in variables. If they can't find any source code tricks, they can still compile and run your tests.

If the students have access to the byte code of your test, they can probably disassemble it and use source code tricks. Even if you effectively obsfuscated it, they can still run it.

If the students know that you are testing student code against other student code, all they have to do is be consistent.

Recommendations

1. Do not make the test available to students. They should be independently testing their homework. After they have submitted their homework to you for formal evaluation, run the tests.
2. Use randomization to make your tests more robust.
3. If your test compares student code against other student code, then you should manually investigate every failure. Maybe one student implemented correctly and the others copied an incorrect implementation. In this case, majority rules rewards incompetence.
4. You may collect their tests and test their tests.