How do commercial Java static analysis tools compare with the free ones?

Question

I'm familiar with a handful of the free static analysis tools available for Java, such as FindBugs and PMD. What I'd like to know is how the commercial products such as Klocwork and Coverity stack up against these. What are their strengths and weaknesses?

Answer 1

here's a list of commercial analysis tools : http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis#Java_2

coverity has several tools :
http://www.coverity.com/html/coverity-readiness-manager-java.html : this should be on par with findbugs and PMD but with better presentation

prevent : http://www.coverity.com/html/prevent-for-java.html : low FALSE POSITIVES.

thread analyzer : http://www.coverity.com/html/coverity-thread-analyzer-java.html : this is what is absent in most open source tools.

Answer 2

The one feature you will most certainly find in a commercial static analysis tool (and that you will not find easily in a freeware analysis tool) is
Reporting: Measures software quality trends over time

As explained in this question about code metrics, any static code analysis in itself in not always meaningful, because you could have:

• too many "defects" to fix
• too many categories of defect reported

You need the ability to do some triage, and you need to check if a particular defect is occurring less and less over time or not, in order to help you prioritize what to fix.

This is especially true on legacy project with thousands of classes: you do not fix defect on many files just like that, without having a good reason. That reason can be deduced from a good reporting and trend analysis you will not find with freeware tools.

Answer 3

I'll suggest you to try SONAR an open source software quality management tool, dedicated to continuously analyze and measure source code quality. This soft take the result from code analysis tool, consolidate that results and give you access to an user friendly interface.

Answer 4

We use a suite of open source and commercial static analysis tools. The different tools find different kinds of bugs and some are tuned for lower false positive rates, at the expense of possibly missing some real problems.

In my experience, Findbugs does a good job of finding real problems, especially if you focus on Correctness errors as their team suggests. Recently the developers of Findbugs have added some basic security vulnerability checks as well. Coverity has a low false positive rate especially if you don't turn on their experimental checkers, and Coverity Prevent includes a good tracking database for trend/cluster analysis. I am not convinced yet that their threading checkers (static or dynamic) work - at least they haven't found anything interesting for us. Klocwork Developer for Java returns higher false positives, but we find they have the strongest security checking of these tools. So it depends on whether your priority is quality checking (Findbugs, Coverity) or security vulnerability analysis (Klocwork, or Fortify). Some of our developers also use PMD to support source code reviews, as it helps with general code cleanup.

A recent project conducted with NIST called "SATE: Static Analysis Tool Exposition" reviewed a wide variety of different tools and their underlying approaches. https://samate.nist.gov/index.php/SATE.html and other references to this project such as at OWASP. The general finding is that different tools have different strengths and weaknesses, so use more than one if you want to do a thorough job.

Answer 5

I have not had direct experience with Findbugs or PMD but have met plenty of people who have compared them with Klocwork and Coverity.

My general take on the feedback has been:

Findbugs and PMD are more "tool-ish". The type of thing you'd run on your desktop. It finds a wide range of potential problems but tends to be noisy, meaning false positives and "I don't care" varieties. It does find some good stuff. I've heard mixed feedback on its long term use. Some feel that the ROI on a free tool is infinite however there is a true cost to false positives.

Not surprisingly, Klocwork and Coverity, which cost money, tend to be more solution oriented that can also scales better to work with teams, has a more efficient, easier to use UI and tends to be less noisy. It seems their analysis is doing deeper inspection and therefore coming up with better results if you did a side by side comparison. When adopting a tool across a team, you'll have various levels of enthusiasm for using a tool and the noise factor is a big issue that prevents widespread adoption. Of course there are things like having support to back you up, etc.

In general, because Findbugs and PMD are free, you see that as a first option. Many companies see value and choose Coverity or Klocwork for a longer term solution although I see also running Findbugs and PMD. They tend to find different things and so if your goal is to find and fix as much as possible, it's good to have a combination of both.

Disclosure: I work for Code Integrity Solutions (codeintegritysolutions.com) which is a partner of Coverity.