Compiler test cases or how to test a compiler

Question

Compilers like all software, would also be prone to bugs, logical errors.

How does one validate the output generated by the compiler. Typically, my question is(are)

• How to validate that the machine code generated is correct?

• How to ensure that the machine code generated is according to the language specification.

• Does it make sense to just pick an open source project (in C if one is also writing a compiler in C) to just compile it through the "compiler". In that case also, how do judge that the compiler is behaving as expected.

• Are there any formal test cases (literature) provided by the language standards committee that a "language complying" compiler has to satisfy?

• What are the sure "give aways" that the problem in a program compiled by a compiler is a compiler bug and not a program bug.

  - Any examples where mainstream compilers get confused and compile the code wrong?

Links to any literature would be appreciated.

Answer 1

There are several compiler test suites out there. We've had some luck using the Plum Hall test suite for a C compiler. It consists of a large set of C code specifically written to test against the language standard. It verifies that the compiler can handle the language syntax and semantics.

Answer 2

The general practice is to create a large set of small programs that each demonstrate one aspects of the compiler. These will include both program that compile and ones that shouldn't. General the ASM coming out the back end is not checked but rather the program is run and it's output checked. As for how to make sure there are no bugs in the test cases: make them small, as in 5-10 lines each.

These test suites can be very large as in hundreds to thousands of tests (for example: an out of date test suite for the D programming language) and generally include one or more test cases for every bug ever reported.

Answer 3

There was an earlier question related to this for C, but it comes down to a carefully written compiler test suite.

As to when compilers get the code wrong, I've hit that often enough in my professional career, thanks. It's happened less and less over time, but I found a bug in MS C++ compilers targeting CLI this week.

Answer 4

The Eiffel compiler is open source and has an extensive library of test cases and internal design contracts.

http://dev.eiffel.com

Answer 5

Good test suites for real languages are expensive to create and maintain. There's a reason that the Plum Hall test suite, which is industry standard for ANSI C, is so bloody expensive.

George Necula's translation validation is a brilliant idea but also quite expensive to implement.

The one thing that's cheap and easy is this: maintain a suite of regression tests, and every time you fix a bug in your compiler, put a suitable test into your regression suites. With compilers, it's unbelievable how easy it is to keep reintroducing the same bug over and over. Disciplined additions to your regression suite will prevent that, and they don't cost much.

Answer 6

For the idea to compile a big open source project:

You could take a project that itself has a test suite. Then you compile the project and its test suite and see if the tests pass. To validate these results you compile project and test suite with an other compiler, and run the tests again.