Unit testing for compiler errors

Question

How do you test for wanted raised compiler errors in unit testing?

Consider the code:

class ErrorTest
{
    OtherClass& read_write() {
        return other;
    }

    const OtherClass& read_only() const {
        return other;
    }

    private:
        OtherClass other;
};

How can I test for read_only() assignment? It's really important and should be firmly checked to properly generate compiler errors:

ErrorTest test;
OtherClass other = test.read_only();
test.read_write() = other.modify();
test.read_only() = other.modify(); /* This should error */

Answer 1

I guess the main question now is, are you testing that your code or the compiler at this point?

Testing the compiler isn't necessarily a bad thing... I've had compiler upgrades mask errors in the past, so it would be nice to ensure that you are getting the same set of safety checks that you expect.

However, you'll have to do a lot of legwork. Your unit test will have to spawn the compiler, capture it's output, and parse it for the correct error statement on the correct line. It's not trivial, and arguably not worth it.

A slightly easier approach might be to keep a directory of bad code, and have a script compile each file one at a time. Have an '#ifdef MAKEFAIL' flag in there that turns on the exact condition that should fail. Ensure the compiler returns 0 when you don't set that flag, and non-zero when you do. That assumes that the compiler returns non-zero on failure... I don't know if MSVC follows that rule.

A third option I'll throw out there, to address portability, is autoconf. It can be a pain to set up, but part of its purpose was to ensure that you have a sane development environment before compiling. You could add a test like this in it, and it let handle finding the compiler and trying it.

Answer 2

This seems a little like the automatic detection that happens when you "./configure" for a build from source on a *nix machine. The autoconf scripts build little programs and attempt to compile them to determine what's available and supported by your compiler.

It probably isn't practical to reuse any of that, but you might want the same model. Each test would have its own file or set of files, and a separate project file/make target/etc.. Then your test script would attempt to make each test case and check that the expected error occurred, either via grep or by comparing the output to a baseline output stored with the test cases.

Answer 3

A valid test of the compiler and platform would be for runtime variations in behavior. I often do this, reproducing small bits of behaviors that exist within my production code, and checking that the behavior is as expected.

But for this case what you really want is a static analysis tool, that checks to make sure the code that your developers write follows conventions such as this one. That is certainly a valid build tool. However, searching for a static analysis tool to find what you're specifying here might be difficult. I'd start with the Wikipedia article List of tools for static code analysis, in the C/C++ section.

Note that the code you've written isn't an "error", it's merely your convention. It's a very good convention, but the compiler does not and should not constrain you to that convention. And neither should a static analysis tool. Therefore you'll need to find one that allows you to configure what is allowable and what is not.