Is there a list of famous software products that do and do not do testing?

Question

I would be interested in looking at a list of projects that did and did not do unit testing, and other forms of regression testing, to see how those companies turned out.

All test infected developers know it saves them time, but it would be interesting to what correlation there is between code quality/test coverage and business success. Something objective like:

xyz corp, makes operating systems, didnt test, makes $50M 123 corp, makes operating systems, does test, makes $100M

Does anyone know of any studies done?

Answer 1

Perl is big on testing and regression testing.

Answer 2

Yes, pick up a copy of Code Complete or even Rapid Development by Steve McConnell. He cites a number of studies.

Answer 3

Any realistic study would have to include thousands of companies. There are far too many factors other than does/doesn't unit test that affect the bottom line. I doubt Microsoft's profit changes all that much whether or not they release an amazing OS every year or one that's as buggy as hell. Just listing a few companies is anecdotal evidence.

Answer 4

I always associate Unit testing with Agile development (XP in particular); you might find that any link between project success and unit testing is influenced by use of agile as well.

I don't know of any surveys specifically, but I did find this just now: http://people.engr.ncsu.edu/txie/testingresearchsurvey.htm which has around 30 llinks to stuff such as: "Qualitative methods in empirical studies of software engineering. Seaman, C.B, Software Engineering, IEEE Transactions on , Volume: 25 , Issue: 4 , July-Aug. 1999"

Not wanted to sound rude - I assume you've already done bit of a search online?

I seem to remember that Code Complete might have references to research into unit testing and project success - but I'm not sure.

Another option would be to approach some software testing companies and see if they had any useful data.

Answer 5

Microsoft commissioned this internal study not so long ago. It compared teams that did and didn't use TDD. To quote the summary:

Based on the findings of the existing studies, it can be concluded that TDD seems to improve software quality, especially when employed in an industrial context. The findings were not so obvious in the semiindustrial or academic context, but none of those studies reported on decreased quality either. The productivity effects of TDD were not very obvious, and the results vary regardless of the context of the study. However, there were indications that TDD does not necessarily decrease the developer productivity or extend the project leadtimes: In some cases, significant productivity improvements were achieved with TDD while only two out of thirteen studies reported on decreased productivity. However, in both of those studies the quality was improved.