c++ continuous integration with performance metrics

Question

I want to set up a continuous integration and test framework for my open source C++ project. The desired features are:

1. check out the source code
2. run all the unit and other tests
3. run performance tests (these measure the software quality - for example how long does it take the system to complete the test)
4. produce a report based on 3. and 4. daily
5. archive the reports for future reference

To implement this, which test framework and what continuous integration process would you recommend? Right now I am leaning towards Google Test Framework (I am aware of some of the comparisons of unit test frameworks discussed in other questions) for tests and Cruisecontrol for continuous integration. But I don't know if Cruisecontrol allows easy integration of performance metrics.

Edit: To answer Wilhelmtell, code should work with both Windows and Linux.

Answer 1

Your question is twofold. As you pointed out yourself, the choice of a unit-testing library is one question. Yes, I think GoogleTest is just fine. I like it because it feels lightweight in terms of usage. UnitTest++ is even more so. It doesn't matter which one you pick.

I don't have any experience with CruiseControl, but looking at your set of requirements I'd probably write a script to do the job. Each requirement can be easily done by at least one tool, so just glue the tools together.

Unless your reports need to be in HTML, XML, LaTeX or other fancy format, I'd use Bash. Then you can just time commands, sed out output from the unittest binary and so on. Still, you might pick Bash for the more complex formats anyway if you really love Bash and are used to it. For instance, you might write an HTML template for the report using placeholders, and then replace the placeholders with the data.

If your tasks are more complex, for instance because you want the script to run on other platforms too then I'd probably use Ruby or Python. In Ruby, for instance, you might use Markaby to output your HTML. Either way, either Bash, Ruby or Python are excellent for text processing or for this task in general. I think you could mock up a small script that would do the work in no-time.

What's nice is that if you use a script then you can also attach it as a hook to almost any modern version control system. For instance, you might set up a Subversion or Git hook such that the script will run upon any commit.

Answer 2

I prefer CruiseControl.Net with cppUnit, but I don't have any experience or knowledge of the google test framework. However, for easy and free continuous intergration you cannot beat CruiseControl.NET especially with all of the testing frameworks, build systems, and other ad dons that it supports.

Answer 3

I am using CruiseControl and UnitTest++ today for exactly this task.

UnitTest++ is really nice although I feel sometimes limited by it around the corner. At least it is 10 times better than cppunit. Still haven't tried the google testing framework, it will be for my next project.

I have been extremely disappointed by CruiseControl. There are a few bugs and questions asked on the mailing list were never answered. Many of the default "features" to manage program execution and logging were not satisfying. We had to write our own report generation, our own stdout and stderr capturing, our own status mailer. There is not much left for CruiseControl, apart from running test at regular intervals and gathering them in a central web server.

Answer 4

We use a combination of NAnt, Cruise Control.NET, bjam and CppUnit and it all works pretty well.

However, it's only quite recently that we've shifted from Windows to Windows and Linux. Currently only bjam and CppUnit are being used on both platforms.

Our NAnt scripts won't be a problem to port; NAnt runs great under mono. CC.NET maybe less so - we haven't gone down that path but there's not a lot of documentation to get it running under mono/Apache...

We're considering replacing CC.NET with Hudson; it's a very promising looking system. Cruise Control is also very good.

We've also considered moving from CppUnit to another test framework but right now it's doing the job fine. GoogleTest (especially with the new mock library), UnitTest++ and Boost Test are all being investigated.

So I know that may not be a conclusive answer but hopefully there's something useful in there for you!

Answer 5

You might also want to ask this question on the CruiseControl user mailing list to see if others are doing the same thing.

But I don't know if Cruisecontrol allows easy integration of performance metrics.

There are two ways you might handle your performance metrics with CruiseControl:

1. If you have xml data you can merge it into the log file then integrate it into email and web pages with some .xsl, or
2. If you have an html report you can use the artifact publisher to make it available from the web interface using the artifact publisher.

I'm a bit biased on the subject because I work on the CruiseControl project but there are several other CI tools that would work as well.

Answer 6

We use UnitTest++ and Hudson.

I find Hudson very easy to use and extremely configurable, with plenty of plugins.