Monitoring code metrics in a multi-language product

Question

We've got a product that's made up of C++ and Java parts. The C++ stuff is build using make and the java projects are made up of some ant projects and some maven2 projects.

I'm looking for a tool that will help me get useful metrics out of the build system over time. examples include

* Total build time
* C++ project build time
* Java build time
* Number of compiler warnings
* Number of unit tests (run/passed/failed/errors). (Tests are written in cxxTest and JUnit)
* Acceptance test metrics (run/passed/failed/errors)
* Total number of files
* LOC (to keep the managers happy)

There's probably loads of other metrics I could think of, but you get the idea.

Getting these metrics for a once-off report is pretty simple. What I really need is a simple tool that will let me plot these metrics over time.

A simple use case where this would be pretty useful would be compiler warnings as we could see the number of warnings trending towards zero over time. (we can't fix them all at once as it's a pretty big project and we just don't have the time for a big-bang approach). It would also help us quickly spot new warnings as they're introduced.

I've seen this question http://stackoverflow.com/questions/1460571/monitoring-code-metrics-in-java-over-longer-time-period, but I'm looking for something a little more language agnostic

So, to sum up. I'm looking for something that reports metrics over time, that's easily extensible, has a web-based reporting gui and preferably cheap. (not asking for much huh!)

Edit: Just to be clear, we're using CruiseControl as our CI server. I just haven't seen an easy way to add metrics or time-based metrics to it's output. Maybe I'm missing something obvious. I've seem this page about adding custom metrics, but it's a little clunky for me.

Ideally I'd love to write out the metrics to a file in a simple format and have something generate the metrics dynamically. Ideally I'd like to turn something like the output below into a simple chart

Build Id | Build Time | Metric       | Value 

00000001   10:45        TestPassRate   95

00000001   10:45        BuildTime      300

00000001   10:45        C++BuildTime   200

00000001   10:45        JavaBuildTime  50

00000001   10:45        TestTime       50

00000002   11:45        ......

Answer 1

You're probably looking for a CI server (continuous integration). These servers watch a source repository (CVS, Subversion, whatever) and build all projects that have changed plus all dependent projects.

In our place, we use TeamCity but there are lots more (list on wikipedia)

[EDIT] Most CI servers show some kind of report after the build (how long it took, how many tests ran, etc). All you need to do is trigger a program after the build which fetches this info and saves it in a database.

You could harvest the historical build pages of the CI server but they usually don't reach very far back which is why it's better to save the data in a different place. If you're looking for something simple for the harvesting, try Python with Beautiful Soup. Otherwise, Java and HTTP Client plus jTidy is the way to go.

Answer 2

If you're using the Java CruiseControl you can get the kind of metrics you want easily. You can include arbitrary .xml in the log file with and then reference any of the values in the reporting .jsp pages. That's exactly how the trend chart for PMD, and checkstyle and Javadoc errors is done. From metrics.jsp:

<jsp:useBean id="xpathData" class="net.sourceforge.cruisecontrol.chart.XPathChartData" />
<%
    xpathData.add("CheckStyle", "count(/cruisecontrol/checkstyle/file/error)");
    xpathData.add("PMD", "count(/cruisecontrol/pmd/file/violation)");
    xpathData.add("Javadoc", "count(/cruisecontrol/build//target/task[@name='javadoc']/message[@priority='warn' or @priority='error'])");
%>
<cewolf:chart id="chart" title="Coding violations" type="timeseries"  xaxislabel="date" yaxislabel="violations">
    <cewolf:data>
        <cewolf:producer id="xpathData">
          <cewolf:param name="build_info" value="<%=build_info%>" />
        </cewolf:producer>
    </cewolf:data>
    <cewolf:chartpostprocessor id="xpathData" />
</cewolf:chart>
<cewolf:img chartid="chart" renderer="cewolf" width="400" height="300"/>

You can just paste this into the metrics.jsp replace the xpath queries w/the xpath to your metrics and you're good to go.

Answer 3

rrdtool might provide the kind of historical view you're looking for. You just need to get your CI server to dump a build report in the right place each time it runs, and rrdtool can take it from there.