How can I simplify testing of side-effect free methods in Java?

Question

Functions (side-effect free ones) are such a fundamental building block, but I don't know of a satisfying way of testing them in Java.

I'm looking for pointers to tricks that make testing them easier. Here's an example of what I want:

public void setUp() {
   myObj = new MyObject(...);
}

// This is sooo 2009 and not what I want to write:
public void testThatSomeInputGivesExpectedOutput () {
   assertEquals(expectedOutput, myObj.myFunction(someInput);
   assertEquals(expectedOtherOutput, myObj.myFunction(someOtherInput);
   // I don't want to repeat/write the following checks to see
   // that myFunction is behaving functionally.
   assertEquals(expectedOutput, myObj.myFunction(someInput);
   assertEquals(expectedOtherOutput, myObj.myFunction(someOtherInput);

}


// The following two tests are more in spirit of what I'd like 
// to write, but they don't test that myFunction is functional:
public void testThatSomeInputGivesExpectedOutput () {
   assertEquals(expectedOutput, myObj.myFunction(someInput);
}

public void testThatSomeOtherInputGivesExpectedOutput () {
   assertEquals(expectedOtherOutput, myObj.myFunction(someOtherInput);
}

I'm looking for some annotation I can put on the test(s), MyObject or myFunction to make the test framework automatically repeat invocations to myFunction in all possible permutations for the given input/output combinations I've given, or some subset of the possible permutations in order to prove that the function is functional.

For example, above the (only) two possible permutations are:

• myObj = new MyObject();
• myObj.myFunction(someInput);
• myObj.myFunction(someOtherInput);

and:

• myObj = new MyObject();
• myObj.myFunction(someOtherInput);
• myObj.myFunction(someInput);

I should be able to only provide the input/output pairs (someInput, expectedOutput), and (someOtherInput, someOtherOutput), and the framework should do the rest.

I haven't used QuickCheck, but it seems like a non-solution. It is documented as a generator. I'm not looking for a way to generate inputs to my function, but rather a framework that lets me declaratively specify what part of my object is side-effect free and invoke my input/output specification using some permutation based on that declaration.

Update: I'm not looking to verify that nothing changes in the object, a memoizing function is a typical use-case for this kind of testing, and a memoizer actually changes its internal state. However, the output given some input always stays the same.

Answer 1

It sounds like you're attempting to test that invoking a particular method on a class doesn't modify any of its fields. This is a somewhat odd test case, but it's entirely possible to write a clear test for it. For other "side effects", like invoking other external methods, it's a bit harder. You could replace local references with test stubs and verify that they weren't invoked, but you still won't catch static method calls this way. Still, it's trivial to verify by inspection that you're not doing anything like that in your code, and sometimes that has to be good enough.

Here's one way to test that there are no side effects in a call:

public void test_MyFunction_hasNoSideEffects() {
   MyClass systemUnderTest = makeMyClass();
   MyClass copyOfOriginalState = systemUnderTest.clone();
   systemUnderTest.myFunction();
   assertEquals(systemUnderTest, copyOfOriginalState); //Test equals() method elsewhere
}

It's somewhat unusual to try to prove that a method is truly side effect free. Unit tests generally attempt to prove that a method behaves correctly and according to contract, but they're not meant to replace examining the code. It's generally a pretty easy exercise to check whether a method has any possible side effects. If your method never sets a field's value and never calls any non-functional methods, then it's functional.

Testing this at runtime is tricky. What might be more useful would be some sort of static analysis. Perhaps you could create a @Functional annotation, then write a program that would examine the classes of your program for such methods and check that they only invoke other @Functional methods and never assign to fields.

Randomly googling around, I found somebody's master's thesis on exactly this topic. Perhaps he has working code available.

Still, I will repeat that it is my advice that you focus your attention elsewhere. While you CAN mostly prove that a method has no side effects at all, it may be better in many cases to quickly verify this by visual inspection and focus the remainder of your time on other, more basic tests.

Answer 2

If you are trying to test that the functions are side-effect free, then calling with random arguments isn't really going to cut it. The same applies for a random sequence of calls with known arguments. Or pseudo-random, with random or fixed seeds. There's a good chance are that a (harmful) side-effect will only occur with any of the sequence of calls that your randomizer selects.

There is also a chance that the side-effects won't actually be visible in the outputs of any of the calls that you are making ... no matter what the inputs are. They side-effects could be on some other related objects that you didn't think to examine.

If you want to test this kind of thing, you really need to implement a "white-box" test where you look at the code and try and figure out what might cause (unwanted) side-effects and create test cases based on that knowledge. But I think that a better approach is careful manual code inspection, or using an automated static code analyser ... if you can find one that would do the job for you.

OTOH, if you already know that the functions are side-effect free, implementing randomized tests "just in case" is a bit of a waste of time, IMO.

Answer 3

I'm not quite sure I understand what you are asking, but it seems like Junit Theories (http://junit.sourceforge.net/doc/ReleaseNotes4.4.html#theories) could be an answer.

Answer 4

In this example, you could create a Map of key/value pairs (input/output) and call the method under test several times with values picked from the map. This will not prove, that the method is fuctional, but will increase the probability - which might be sufficiant.

Here's a quick example of such an additional probably-functional test:

@Test public probablyFunctionalTestForMethodX() {
   Map<Object, Object> inputOutputMap = initMap(); // this loads the input/output values
   for (int i = 0; i < maxIterations; i++) {
     Map.Entry test = pickAtRandom(inputOutputMap); // this picks a map enty randomly
     assertEquals(test.getValue(), myObj.myFunction(test.getKey());
   }
}

---

Problems with a higher complexity could be solved based on the Command pattern: You could wrap the test methods in command objects, add the command object to a list, shuffle the list and execute the commands (= the embedded tests) according to that list.

Answer 5

Im afraid i dont find the link anymore, but Junit 4 has some help functions to generate testdata. Its like:

pubic void testData() {
  data = {2, 3, 4};
  data = {3,4,5 };
...
 return data;
}

Junit will then thest your methods will this data. But as I said, I cant' find the link anymore (forgot the keywords) for a detailed (and correct;) example.

Answer 6

have a look at http://fitnesse.org/: it is used often for Acceptance Test but I found it is a easy way to run the same tests against huge amount of data

Answer 7

In junit you can write your own test runner. This code is not tested (I'm not sure if methods which get arguments will be recognized as test methods, maybe some more runner setup is needed?):

public class MyRunner extends BlockJUnit4ClassRunner {

    @Override
    protected Statement methodInvoker(final FrameworkMethod method, final Object test) {
        return new Statement() {
            @Override
            public void evaluate() throws Throwable {
                Iterable<Object[]> permutations = getPermutations();
                for (Object[] permutation : permutations) {
                    method.invokeExplosively(test, permutation[0], permutation[1]);
                }
            }
        };
    }

}

It should be only a matter of providing getPermutations() implementation. For example it can take data from some List<Object[]> field annotated with some custom annotation and produce all the permutations.

Answer 8

I think the term you're missing is "Parametrized Tests". However it seems to be more tedious in jUnit that in the .Net flavor. In NUnit, the following test executes 6 times with all combinations.

[Test]
public void MyTest(
    [Values(1,2,3)] int x,
    [Values("A","B")] string s)
{
    ...
}

For Java, your options seem to be:

• JUnit supports this with version 4. However it's a lot of code (it seems, jUnit is adamant about test methods not taking parameters). This is the least invasive.
• DDSteps, a jUnit plugin. See this video that takes values from appropriately named excel spreadsheet. You also need to write a mapper/fixture class that maps values from the spreadsheet into members of the fixture class, that are then used to invoke the SUT.
• Finally, you have Fit/Fitnesse. It's as good as DDSteps, except for the fact that the input data is in HTML/Wiki form. You can paste from an excel sheet into Fitnesse and it formats it correctly at the push of a button. You need to write a fixture class here too.