C++ equivalent to Python's doctests?

Question

I think the concept of Python's doctests is brilliant, and as a C++ programmer at a real-time shop, I'm quite jealous. We basically have no unit test capability, which is a severe hindrance. I've seen C++Unit, etc, but is there anything that can extract test cases out of comments like Python's doctests rather than putting them in the code directly?

Answer 1

I have just had a look at doctest, it is indeed brilliant. So is the ease of use approach of the Python language.

For C++ however, you probably won't find such a tool.

If one of the tests fails you might want to debug what happens. It wouldn't be that easy having the source of the test case been generated from comments. Instead, in the existing unit test frameworks for C++ you have the source of the tests with a special syntax, which is compiled and is easy to debug. Also, the generation of the source from comments would be another extra (pre-)compilation step, which would just make life harder.

You should accept that coding in C++ is not as a quick job as it is in Python, so the unit testing is similarly somewhat more difficult. On the other hand you have more tools, you can put static assertions for type relationships, for example, which is impossible in Python.

So briefly I think the analogue of the Python doctest for C++ would have so many drawbacks compared to the existing tools, that nobody began to implement it.

If you really believe that it can be better than the existing tools, please provide some examples. I hardly believe that there are real life cases where it can be more usable.

Answer 2

I was thinking something along the lines of generating CxxTest files from comments. I haven't used that framework, but it looks promising. From their manual, a unit test file looks something like this:

 // MyTestSuite.h
 #include <cxxtest/TestSuite.h>

 class MyTestSuite : public CxxTest::TestSuite 
 {
 public:
    void testAddition( void )
    {
       TS_ASSERT( 1 + 1 > 1 );
       TS_ASSERT_EQUALS( 1 + 1, 2 );
    }
 };

My proposal would be a parser that extracts the contents of those testX functions from comments, rather than having to write the whole thing. For example (and I'm just making up the comment syntax here, there may be a cleaner way to write it):

// MyRegularCode.cpp

/// Description of the function here
/// Then test case below that gets extracted
/// and turned into CxxTest .h files
/**testAddition
MyClass mc;
mc.MyFunction();
TS_ASSERT( mc.m_value > 1 );
TS_ASSERT_EQUALS( mc.m_value, 3 );
**/
void MyClass::MyFunction()
{
    m_value = 3;
};

I'm not sure how the more powerful aspects of CxxTest would get implemented, such as creating fixtures, but something like this might provide the together-ness of python docstrings and doctests in the C++ world.

Answer 3

I'm aware of old tricks with putting things in real code comments (IIRC, this is part of Practice of Programming). However, it may be easier to simply put the unit tests in #ifdef blocks. You can generally run the preprocessor by itself for this sort of thing.

Then again, I'm aware of one project that uses Perl as a super-duper preprocessor.

Answer 4

The test framework that comes with Fost does handle something fairly similar. The tests would not be embedded in the documentation, but they can sit alongside the code that they test. In structure the tests look very similar to the cxxtest code.

#include "myclass.hpp"
#include <fost/test>

FSL_TEST_SUITE( myclass );

/*
    Your documentation
*/
FSL_TEST_FUNCTION( constructors ) {
    fostlib::test::default_constructable< myclass >();
}
myclass::myclass() {
}

FSL_TEST_FUNCTION( some_method ) {
    myclass instance;
    FSL_CHECK_NOTHROW( instance.some_method(0) );
    FSL_CHECK_EQ( instance.some_method(2), 2 );
    FSL_CHECK_NEQ( instance.some_method(-2), 0 );
}
int myclass::some_method( int arg ) {
    // implementation
}

All of this lot gets compiled with the tests embedded (you could remove them from the build via a #define -- not implemented, but easy to do). The tests are then run through a separate program that loads the .DLL or .so that has been built, finds the tests and runs them.

We've not tried it, but it should work with static libraries and dynamically loading and running tests found in .EXE files on Windows, but I'm less sure if it can be done like that on Linux or Macs.

Answer 5

You might find this useful. Ive started developing this after needing this in my own code.

http://github.com/panyam/DocTestPlusPlus

It is a python script that goes through your comments and extracts tests and generates test files.

Still under development and testing. Appreciate any and all feedback.

cheers Sri