How popular are security code reviews?

Question

Real world, how many here undergo in-depth security code reviews? Those that do, how often - once a quarter, once a version, once a blue moon? Those that don't - why not? (Not referring to small or hobby programmers - not that I'm trivializing them, its just I don't expect them to ;-) ).

As a security consultant, I'm usually the one called in to do the security reviews, however this is usually only either for very security-sensitive organizations (e.g. large banks, software vendors, military, etc), or as a result of regulatory requirements (e.g. PCI-DSS).
Now, few groups (except those in the biggest companies such as Microsoft, Intel, RSA, etc) really enjoy the review, even though it really should be a positive experience. It seems to me that this is mostly because of the perceived high investment - of resources, time, and of course cash to bring in the consultants.

Okay, so it's not just perceived, it's real enough: it's commonly accepted that a single reviewer can cover between 50-100 LoC per hour. Though we've managed to multiply that - since we're only looking for specific security issues (and the clients are pressing hard for lower costs) and we can minimize the scope according to the risk - we can max out at around 1000 LoC per hour. For any medium-to-large system, this is still hundreds of costly consultant hours, not trivial at all.

The common suggested alternative is automatic source code scanners, ala Fortify, ounce labs, etc. However, besides the licensing costs, this is far from efficient - typically we find these tools to produce results in the 100 thousands, with a very high (70-90%) rate of false positives (and duplicates). So you're still spending large chunks of time going over the results, AND these tools do not cover a substantial set of potential vulnerabilities (e.g. logic flaws, business logic, etc)

That said, (and a big DISCLAIMER should go here:) I've been working the past few months with one of the tool vendors to develop a service that would do this very efficiently - e.g. be able to cover 500K LoC in just around a single week of work, and yet provide actual, real accurate and complete results - virtually ZERO false positives and nearly no missed false negatives.

Those of you who should be doing SCR, but aren't - would this be enough to convince you otherwise? or is there something else holding you back? Or is it just not an issue for you?

---

To clarify, I'm not trying to promote myself or my service, just trying to get some real-world perspective beyond my own security-evangelistic agenda. I'd like to see the issues as other programmers see them...

Further clarification, I am NOT asking HOW to perform code reviews, what options there are, etc. I have much expertise in this field, and it is this expertise that I sell to my clients. My question is IF code reviews are as unpopular as they seem, WHY this specific activity is not as popular as it should be, and HOW we can go about changing this. (Irrelevant of choice of methodology, tool, etc.)

---

Furthermore, as Corneliu and others pointed out, security code reviews should NOT be taken alone as the sole protection and verification of a system's security, rather should be one element of a complete, holistic SDLC (secure development lifecycle) framework. However, neither should it be forgotten. So my question is really focusing on that one element, whether in the context of the full SDLC or alone as a step forward from penetrate-and-patch.

Answer 1

To be honest, rarely if ever have them.

As much as the business is aware that security is important it is always one of those things (much like testing) that ends up taking a hit in favour of getting in new features, changes etc.

Whilst everyone knows this is the wrong way to go about things I doubt much will change until the application gets bitten badly by a security vulnerability. Sorry for being so cynical.

Answer 2

Hi AviD,

I'm from Australia and I have to say that security reviews are (maybe) more popular here.
In the last two projects I've worked we had formal security reviews. I have one more review aligned in the close future but from my perspective a "security review" is wrong by design.

People who do security reviews just look for a cheap silver-bullet to tell them the code is not completely busted and that the chance for an application attack is slightly lower than "it will happen in the first 10 hours of production".

I think a security review starts with:

• an Architectural Review from a Security Perspective when the architecture is put in place.
• Then you work with the architect + business on Threat Modelling and understanding the potential issues to be faced and mitigations that can be put in place.
• Then you build an approach for putting these mitigations in place preferably at an architecture level and try to isolate business-level developers from as much of the security code as possible.
• Next I would do the full Security Development Life-cycle with the team and teach them proper secure-coding guidelines. Show them all the types of attacks that can happen on their application based on the Threat Model that you built before. Show them how to code and protect.
• Teach the team about defence in depth and how to secure each piece of their puzzle
• Teach them how to deploy securely, how to design isolated pieces with a minimum required security on them and so on.
• Teach them how to do security code reviews of their own code and provoke them to provoke each-other in writing better code.

This would take you a good two weeks I'd say. Maybe three.

Then, you come back regularly and do partial reviews, update the threat-model, and provide more guidance on potential issues that they might introduce.

With this you have a better chance at the end of the project to do a final code review and have a good trust that what you have is good and secure.

Oh, and you can sell this to a business much better than "I need a month to look at your code base and I'll let you know how busted your app is". You can go in holidays for a month and come back and tell them all is good mate and they have no idea what you did.

My 2 cents,

Corneliu.

Answer 3

It's dependent upon how important the org and the app is.

You may want to check out Veracode and Fortify (contrasting styles) that automate the process. Both decent outfits and both with a decent client list.

This would be used in conjunction with an Application Security Pen Test. If you're presenting a logon to a protected resource over the Internet then you probably want to run through the code making up the Auth Module. You do need to automate it somewhat - small applications can contain 100K lines of code and more. Would you manually sit through all of this? It sounds pretty boring?

You also need to consider their value add. If you're infrastructure is appallingly open then theres little point in running a Code Review. Why check the windows to the house when all doors are open?

At a bank I worked with (in a Security Consulting /Architect /general dogs) as part of their Project lifecycle, each Project would make up the cost of their Code Review Tool (Fortify in this particular instance). However some of their applications were been broken and owned in less than 5 minutes because of poor IIS configuration.

Code Reviews - at what point in the Dev cycle would this be done? It becomes a little more interesting from an Agile perspective.

Noelie Dunne

Answer 4

Running Fortify (or similar tool) against any Enterprise App will always find a shed load of vulnerabilities. How many of these are theoretical, how many of these are real business impacting risks? How do you categorize what's important and what’s not & cutting out the false positives? What do you run Fortify against?

This is the approach I generally take from a Security perspective in Enterprise wide Application’s /Architecture …. First understand the ‘use’ cases for the application and the interfaces/components/coupling into and out of your application. Break this down into the user & system interfaces User - you're looking at the use cases for the App i.e. browser /thick client, levels of privilege System - things like MQ, Tibco interfaces, any Management /Admin, Oracle TNS listener, ODBC, ADO, proprietary interfaces and so on.

If you’re a Trading Application dealing in bonds/options then what are you’re interested in?

1) The ability for an anonymous user to access any private application data 2) The ability for an authorised user to exceed permissions within their profile 3) The ability for an authorised user to access a different user’s data 4) The ability for an authorised user to access different users processing results 5) The ability for an authorised user to gain any level of access to any components of the application or internal architecture

So this will be driven the by the logon module and the components that the module calls following successful auth. You’re primarily interested in performing input validation & code checks around this.

There’s absolutely no value code checking components, dlls that have you are not accessible directly or indirectly through another component.

And this assumes that the platform infrastructure is on the money.

Answer 5

I'm not sure if this is the type of answer you were looking for, but hopefully it might be helpful to you.

I have a couple thoughts. First, I value security highly and think security code reviews are a good idea, but I can see why they might not be popular in all cases, depending on the industry. This is pure speculation, but a lot of companies do public betas and so they rely on the feedback received from those sessions. They may view security as something that should be iteratively improved and so they place less emphasis on it at the beginning, confident that a few smart testers will find the bugs and that the beta label will protect them from any liability. Obviously that isn't all companies, but a few might be thinking along similar lines.

If you really want to expand your market, you might consider pitching it in a different light. In addition to highlighting security reviews as potential risk management, they could be viewed of in terms of cost-savings over time in regards to technical debt, salary, reputation, and legal issues.

For example, I'd rather pay a lawyer up front to write a bullet-proof contract than to represent me in a court of law over a contract dispute. It's a form of risk-management, but I also view it as an investment and an enormous savings over time. Not only am I more secure legally, but there are also intangibles such as a defined relationship between myself and a client, where there are set boundaries and expectations to be met by both parties. With those boundaries in place, you are actually freer to concentrate on the things you need to focus on.

If you can present security in a similar fashion, in terms of savings and peace of mind, you may find that the people (read management) who make decisions about hiring security consultants will be more open to your services because you are affecting their bottom line in a positive way. Present it in such a way that you save them far more than you cost.

Answer 6

Since you are doing a market-research, I'd like to contribute.

I'm doing so much SCR for other companies (generally as you said, banks, big e-commerce apps, critical components and other financial stuff). False-positive and money-wise don't use big guys : Fortify, Ounce Labs.

If you are planning to bring a new tool in that market with low false positive and a decent price tag (because Fortify and Ounce are being ridiculous -since they dominant the market-) I and so many other people can get it. MS is digging the same market in VS.NET area, so some developers might go that way and MS can crush any other opponent in there (at the same time can bring some public attention to the market). You should be careful about that.

Not a direct answer to your question, but I hope it helps and if you release it keep me posted.

Answer 7

Security professionals shouldn't be reviewing code 100% manually. I hope that you have a copy of Ounce Labs or equivalent and are only reviewing to eliminate false-positives, tuning rules, etc.

Answer 8

Your point about time and cost of an external code review is key. My company contracted Cigital a couple of years ago to do a detailed tool-assisted code review. The security analyst used Fortify and spent weeks completing the manual code review. The time and cost, and the time taken to support the analysis, answering questions and interpreting results, was more than we could sustain on a regular basis. We do our own code reviews, using security guidelines from OWASP and checklists prepared by Cigital. And we use static analysis tools (Coverity, Findbugs) in our continuous integration environment, each release is checked with IBM Watchfire Appscan (or whatever IBM calls it today) and we schedule regular pen tests with Foundstone. The pen tests are a good use of money, and the static analysis tools help ensure high code quality.

If there was a cheap, fast and effective security code review offering, yes there would be a market. I build financial software: my customers aren't crazy for the idea of sending software overseas for analysis by Veracode or similar tho.

Answer 9

I participate in an open source project called Gallery. When we hit release candidate 1 for any given release, we hire an external team to do a security review. The goal is to get a fresh set of highly trained eyes who and focus them on any latent security issues that have crept into the product. In any typical release, we have found that the security review (while expensive) turns up issues and classes of vulnerabilities that our typical developers would not have uncovered. Even though we are a reasonably security conscious development team, there are always new attack vectors that we are unaware of that we have to address.

Thus far we've had 3 full security review for the Gallery 2.x products, and 1 security review for the Gallery 3.x product (in that case we did the review while was in alpha because we wanted feedback about vulnerabilities as soon as possible).

Automated (read: static analysis) tools would be nice. Automated pen-testing tools are also nice. But we believe that there is a great deal of value in having a human in the process to investigate the code and look for the subtle vulnerabilities that trip us up.