COTS vs. Custom / Build vs. Buy: Decision Tree and Best Practices

Question

Background:

I work at a company with a large SAP investment, and we also have dozens of large .NET systems (mostly internally for engineering systems), and Java platforms (mostly for external web applications). As such, we have large development shops on ABAP, C#, and JEE.

Question:

In short, We need a better way to determine when Commercial, Off The Shelf (COTS) software should be used, and when we should leverage our own developers.

Criteria:

I'd like to build a decision tree based on best practices to help with this question.

At the highest level, Jeff Atwood's related post sums it up well: The Best Code is No Code At All

A little deeper, I'd like to see criteria like:

Is a COTS system available that meets most of the requirements? (If yes, a COTS system may be a good option: (Avoid reinventing the wheel))

• If so, is there a fully exposed API available? (This is essential to integration / customization)
• If so, is the source code available? (This is essential to deep integration / customization)

Is the system designed to meet a core business function / create a competitive advantage ? (If so, custom development may be a good option: See Joel Sposky's: In Defense of Not-Invented-Here Syndrome)

• If so, will custom development allow for code reuse in future / other systems? (There are many advantages to reusing existing code)

What is the TCO for the custom application vs. the COTS product?

Are there time constraints that could not be met with custom development? (If yes, a COTS system may be a good option)

Answer 1

I'm not entirely sure what you are asking for but I'd thought I'd comment on a few things that I've seen arise in COTS versus custom development choices over the years:

1. It is going to take time to properly analyse any COTS systems for suitability. Both from a requirements perspective and a technical one. How much custom dev could have been done instead of the analysis?

2. Beware the COTS sales pitch the promises the moon on a stick. There's plenty of them. Flashy presentations from yes-men who will offer to meet any requirement to get the deal. The most dangerous trap to fall into is being promised functionality that isn't in the COTS at present but they will add for you - more often than not the salesman has said yes to you without even finding out if it's possible for their product to do it.

3. Check for unit tests in the COTS and also what development practices they use. Good indicators of quality. A cowboy development practice, lack of tests and documentation are maintainability headaches in the future.

4. Be wary if the COTS vendor isn't giving much info about technical aspects of their product.

If your desired system is fairly simple then your COTS choice will also be fairly simple. But if it's a large, complex system you would presumably offer it out for RFP (request for proposal) and to do that you are going to have to have a thorough and correct requirements specification. Will the time taken to produce requirements for the RFP out weigh a custom dev agile solution? You are going to have to nail those requirements down super tight to make sure the COTS system delivers and that will take a lot of time and effort.

Personally, I would never consider COTS unless:

1. The source code is available and I have had programmers assess it
2. I've seen and tried a working demo and not just glitzy sales pitches
3. There isn't time or personnel to do it in-house.

Ultimately, I agree with Joel's statement: If it's a core business function -- do it yourself, no matter what.