Why do many software projects fail today?

Question

As long as there are software projects, the world is wondering why they fail so often.

I would like to know if there is a list or something equivalent which shows how many software projects fail today. Would be nice if there would be a comparison over the last 20 - 30 years.

You can also add your top reason why a software project fails. Mine is "Requirements are poor or not even existing." which includes also "No (real) customer / user involved".

EDIT: It is nearly impossible to clearly define the term "fail". Let's say that fail means: The project was more than 10% over budget and time. In my opinion the 10% + / - is a good range for an offer / tender.

EDIT: Until now (Feb 11) it seems that most posters agree that a fail of the project is basically a failure of the project management (whatever fail means). But IMHO it comes out, that most developers are not happy with this situation. Perhaps because not the manager get penalized when a project was not successful, but the lazy, incompetent developer teams?

When I read the posts I can also hear-out that there is a big "gap" between the developer side and the managment side. The expectations (perhaps also the requirements) seem to be so different, that a project cannot be successful in the end (over time / budget; users are not happy; not all first-prio features implemented; too many bugs because developers were forced to implement in too short timeframes ...)

I',m asking myself: How can we improve it? Or do we have the possibility to improve it? Everybody seems to be unsatisfied with the way it goes now. Can we close the gap between these two worlds? Should we (the developers) go on strike and fight for "high quality reqiurements" and "realistic / iteration based time shedules"?

EDIT: Ralph Westphal and Stefan Lieser have founded a new "community" called: Clean-Code-Developer. The aim of the group is to bring more professionalism into software engineering. Independently if a developer has a degree or tons of years of experience you can be part of this movement.

Clean Code Developers live principles like SOLID every day. A professional developer is the biggest reviewer of his own work. And he has an internal value system which helps him to improve and become better.

Check it out on: Clean Code Developer

EDIT: Our company is doing at the moment a thing called "Application Development and Maintenance Benchmarking". This is a service offered by IBM to get a feedback from someone external on your software engineering process quality etc. When we get the results, I will tell you more about it.

Answer 1

The last statistic that I heard was that 90% of projects are either over time or over budget. So if you consider that failing, quit a bit.

The reason why it fails mainly lies on process. We as software engineers don't do a good job of gather requirements, and controlling the customer. Because building software is a "mysterious" job to people outside of IT, it is difficult for them to understand why last minute changes are difficult. It's not like building a house and clearly showing them why it isn't possible to quickly add another door to the backside of the house made of brick.

Answer 2

Not a direct answer, but I found the Virtual Case File to be a fascinating case study on how a big government-backed well-funded project can still tank.

You can also add your top reason why a software project fails.

Another IEEE Spectrum Online article "Why Software Fails" examines this very question. It summarizes the major points as follows:

• Unrealistic or unarticulated project goals
• Inaccurate estimates of needed resources
• Badly defined system requirements
• Poor reporting of the project's status
• Unmanaged risks
• Poor communication among customers, developers, and users
• Use of immature technology
• Inability to handle the project's complexity
• Sloppy development practices
• Poor project management
• Stakeholder politics
• Commercial pressures

Answer 3

Honestly, I think its because most programmers are not very good at what they do(and I don't mean just cranking out code). People on stackoverflow are probably the exception. I don't know about the rest of you but as a consultant/contract programmer I have worked in or around many places, and the ratio of mediocre or poor programmers to good ones is about 10 to 1.

One of my strengths has always been estimating accurately and then delivering on time and on or under budget - I always aim for coming in 10% under cost and on time. Then I like to tell my client that because I got things done for less $$ than expected, which of the "extras" would you like to add in?

Even a perfectly functioning product that is late and/or over budget will be considered a failure by many business managers. Programmers often focus on just the technical aspects of what they do, with little regard for the cost or deadline. You really need to do all three well for it to be deemed successful project. There are many other programmers that could code circles around me without a doubt, but for the person paying for the project, that is rarely enough.

Answer 4

People/companies do not proudly shout about their failures, so many cases wont be ever heard.

Answer 5

Not only software projects go over budget, or take more than scheduled time to complete. Just open the newspaper and look at public projects like bridges.

But for a software project it is far more easy to cancel everything. If a bridge or building is half finished, there is no way back. Half the infrastructure is in place. It is very visible and it takes money to remove it.

For a software project you can press Shift-Delete and nobody notices.

Its just a very hard job to do an accurate cost analysis.

Answer 6

Poor planning.

Answer 7

Hofstadter's Law

It always takes longer than you expect, even when you take Hofstadter's Law into account.

Answer 8

One common mistake is that sales people and technical people do not communicate sufficiently. So the salespeople sell things that are technically not feasable within budget. (And then they run with their bonus :) )

Answer 9

Using the FBI's Virtual Case File system it comes down to poor program management. The program had pre-9/11 requirements and post-9/11 expectations. Had government management done their job there would have been contract mods.

http://government.zdnet.com/?p=2518

"Because of an open-ended contract with few safeguards, SAIC reaped more than $100 million as the project became bigger and more complicated, even though its software never worked properly. The company continued to meet the bureau’s requests, accepting payments despite clear signs that the FBI’s approach to the project was badly flawed, according to people who were involved in the project or later reviewed it for the government."

Although $105,000,000 for 700,000 lines of code comes to $150 per line of code. Not too bad.

Answer 10

(From a programmers point of view - I'm not a project managemer, but I've often been involved in the process).

A number of people have mentioned that bad programmers are endemic. But I think this is true in another sense as well - we're all bad programmers in that we find it difficult to anticipate complexity, an unavoidable issue that 50 years of magic bullet estimation and planning schemes have failed to solve.

Anticipating the side effects of large projects gets exponentially more difficult as projects grow. This is a dull truism, for sure, but for me it means that on any project I've worked on where I've been involved in the estimating process I've run into some case where there's an unanticipated consequence of a design decision that causes everything to come to a grinding halt, or at least a few days of bugfixing - just something that nobody foresaw, not any sort of malpractice or stupidity. It's just the nature of a complex enough system.

Aside from the built-in uncertainty, there's also a tendency to underestimate things whose outline is known, because the fact that they have less uncertainty makes them seem simpler to implement.

So the uncertain stuff gets magnified, the clear stuff gets minimized, and what really kills you is the thing that you didn't think would be uncertain.

Answer 11

Mismanagement.

SW project get started by throwing developers against a perceived problem. Business requirements crystallize as the project progresses. New functionality gets added while deadlines stay put. More developers are thrown in. Original project members quit or get fired. By this point too much time, money and resources is invested in the project so it cannot be canceled. As the deadline passes the project is declared finished and successful despite the obvious lack of finished product.

Come to think of it - I've jet to see a SW project fail...

Answer 12

Bad management.

Projects are not successes or failures based on some underlying feature of the project, but on whether they fulfill the needs of the users. (They can fail altogether, in which case there was a gross misstatement of what was possible.) It is mostly in the process of evaluating the feasibility and cost-benefit ratio of the project, and establishing goals, that software projects tend to fail or succeed.

There's a disconnect between people who deal with facts and things (like programmers) and people who deal with other people (like sales types and managers). To a programmer, the facts are the facts, and have to be dealt with. To a sales person, the facts are what other people think, and are changeable.

There's also differences between tangible and intangible facts. Nobody thinks that workers could build a large bridge in a month if they were really motivated; they can see all the steel and concrete and other stuff that has to be moved and fixed into position. Software is much less tangible, and lacks the physical restrictions: while it is not even theoretically possible to build the bridge within a month, it is conceivable that a team could create a large project within a month, as "all" they have to do is get everything right the first time. It is physically possible to type thousands of lines of code a day; it's just that the chance that they're usable as is is so close to zero it doesn't matter. The actual productivity of a top developer is actually pretty unimpressive in word count, compared to (say) the productivity of a journalist.

Therefore, those who are used to flexible facts don't have the imposing physical limits to remind them that things can be pushed only so far, no appreciation for what programming actually requires, and no good feel for how much productivity is realistically possible. Moreover, they know how to get their way in negotiations, much more than the average developer, so in negotiations about what's possible they tend to assume more than they can, in the real world, get.

In the meantime, software development is inherently fuzzy, because producing the physical product is trivial. I can produce a copy of software quickly and cheaply, once it's been developed. Software development is design work, pure and simple. Anything corresponding to manufacturing is ruthlessly eliminated with such things as compilers and wizards and code generation. The developer, faced with the manager who wants the impossible, finds it hard to say the impossible is actually impossible, because there's no way to say it's actually impossible. Given facts that are unknown enough to feel flexible, the person with strong negotiating skills and determination will typically get the answer he or she wants.

Given this disconnect, one might ask whose responsibility it is to bridge it. The answer is, in my opinion, clear. The responsibility for understanding how different people think belongs to the people who specialize in dealing with other people. The responsibility for coordinating different types of people belongs to the people whose job it is to coordinate these things. Therefore, managers.

Managers who do understand software development and developers, and can deal well with other managers, will do well, and their projects will generally succeed. There are still far too many of the other type in the world.

Answer 13

Poor use of practices and software development methods. In my experience, one of the big reasons a project failed its that the development team use a wrong method to face the software development process. Choosing a methodology without having a good understanding of how it works and what it takes, can bring a time consuming issues to a project, like poor planning.

Also a common problem is also the use of technologies without a previous evaluation of it to understand how it can be applied, and if it brings any value to the project.

Answer 14

The number one reason: a failure of project management.

A PM's raison d'etre is to make a project succeed, ergo a project failure is their failure. Certainly there are factors beyond their control, but it's still within the PM's job description to manage that risk, and the only get out clauses should be someone higher up the food chain taking decision control (which is a terrible thing to do to a PM) or acts of god.

In my experience failures mostly occur when PM work has been fast and loose or non-existant, including when decisions start to flow from sales people and when the client starts decreeing change control. A good PM is priceless.

Answer 15

It is because no-one seems to read anymore. Every single reason why projects fails has been analyzed time and time again. You only have to read three books to know why 80% of projects fail:

The Deadline: A Novel About Project Management (Tom Demarco, published 1997) It's a great introduction and it's pretty entertaining. Peopleware : Productive Projects and Teams (Tom Demarco, published 1987) The Mythical Man-Month: Essays on Software Engineering (Fred Brooks, published 1975)

We as a profession simply seem to forget everything every 3-5 years (see "centralised computing is inefficient; let the clients handle it" vs cloud computing).

Answer 16

Failure is a judgement -- more of an accusation, really.

"The project was more than 10% over budget and time."

Which budget? Which schedule?

6 months ago, I wrote a plan saying it would take 6 months.

3 months ago, the users asked for more stuff. I gave them a plan that said it would take 9 more months.

Last month I was told that the project was 6 months over budget and therefore a "failure".

But wait. It delivered what the users wanted. It was over the "original" estimate. It was under the revised estimate. The users want more. IT wants less.

Answer 17

To truly gage whether a project is really successful, the original estimate / budget is probably a poor yardstick. Most engineers tend to underestimate how much time it will take because of political pressure, and they don't know how to estimate anyway. Many managers are poor planners because they want unrealistic deadlines to please their boss, they often don't understand what's involved, and plus it's something they can look at and understand and use as a stick in meetings, as opposed to actually helping solve problems. Practically, it helps businesses make rough projections of expense for budgeting purposes, at least it gives them something the go by.

A better measurement of project success would be - are the users happy? Does it help the business make money? How fast will the money gained help recover the cost of the system? These are harder to gauge than a simple plan, though.

In the end, I've find you're better off delivering on deadline, even if it means working some overtime. It sucks, but that is the reality of it.

Answer 18

As stated previously the vast majority of people involved in software development to not actually understand how

• ask the right questions to learn about the problem
• appreciate the users goal and judge expectations
• understand technology available and the related Eco structure
• most of team directly/indirectly involved are poorly skilled.
• do not appreciate or know when they are wrong do they can take action.

Even with perfect requirements and related definitions too many developers don't know what they are doing.

Just look at the types of questions asked here. Would you go to a doctor that asked the same equivalent question. The scarey thing is that they ask and don't know how to learn or reason.

Answer 19

I'll approach it from a different aspect than most the rest here.

I've noticed a project slowly fail over a period of time. Sure, it's gotten better in that time too--but it still isn't profitable. In this market profitability, and being in the black, means success.

Why is it failing? I think it's simple: you get what you pay for.

Software is like a bank account, not primordial ooze. If you don't put resources into it (time, money, focus, effort) then you won't get anything out of it except failure and cost. So you must invest things into your project, and sometimes the earliest work sets the stage for years to come. You can't throw mud at your computer and expect a new mouse in two years and $10 million dollars later, so likewise there must be effort expended.

One of the biggest problems today are "budget developers" in a third-world country. I don't begrudge them their part of the market, but for a well-funded Silicon Valley startup to seek them out and get a budget foundation (framework or even prototype) is to make a poor investment in the future. This very same budget framework is what is causing my friends so much of a hassle today. It works now; it worked when it was written, but it wasn't written well and few even take the time to maintain it. Were the company to stop and rewrite the software the way it should have been written in the first place they wouldn't have all this trouble. Can they afford the time? Nope. They have to make it profitable before they can even thing of it.

As the saying goes, "I can make it: cheap, fast or good. Now, pick any two of those." Everyone wants all three, myself included. But if you don't invest the time, planning, and work required to make your project a success from the start ... then don't expect anything you can be proud of later. It'll feel like a forged Mona Lisa where you, and every other engineer like you, can see a defect here and there that shouldn't have been there from the start.

So:

• Don't undertake what you cannot afford in: time, money, effort, focus, etc.
• Don't skip planning!
• Don't be afraid to rewrite early when it counts the most. (Later it'll be worse than a trip to the dentist, believe me.)
• Don't underestimate the power of bureaucracy to prevent you from doing it right.
• And don't be cheap where you should spend the most of your time. It will cost you later, guaranteed. And if not you, then someone else will take the bullet for you.

Answer 20

There have been some good studies done on this. I recommend this link from the Construx website (Steve McConnells company).

Answer 21

The Construx link above is real good!

Many projects are managed on a rosy picture of reality. Managers tend to power talk developers into optimistic estimates. But say a 20 week projects gets "talked down" to 10 weeks. The requirements phase will now be 1 week instead of 2. After 1 week, the requirements aren't finished, but the project moves on. Now you're working on shaky ground-- and on a stretched schedule!

This can be funny. Once there was this old guy in a room opposite mine. His job title was system adminstrator. But the system he was supposed to adminsiter wasn't there. It had never been finished, although management thought it had been. The guy played games for about a year before he got bored and moved on.

The funniest part? Management put up a new job opening after he left!

Answer 22

Different agendas

Management do not really understand what a developer does, how they produce code and the difficulties encountered. All they care about is the final product delivered on time. But for a developer they care about the technical aspects, well produced code in a solution they our proud of.

Deadlines

I've often heard developers say they wish they could produce better code, and that deadlines often push them into producing something that just works, rather than good code. When these deadlines are unreallistic the problems are exacerbated.

Answer 23

IT Project Failures is a blog about project failures that may have a few answers here if one wants to read about it.

Myself, I think a large part of this lands on the question of being able to state exactly what is expected in x months at $y when really the answer is much more open-ended. For example, if a company is replacing an ERP or CRM system, there is a good chance that one isn't going to get all the requirements exactly right and thus there will be some changes, bug fixes and extras that come from taking on such a large project. For an analogy consider how many students entering high school or university could state their precise schedule for all 4 years without taking any classes and actually stick to that in the end. My guess would be very few do that as some people change majors or courses they wanted to take aren't offered or other things happen that change what the expected result is but how is this captured in a project plan that we started here and while we thought we were going there, we ended up way over in spot number three.

Answer 24

Being over budget and time is not a good definition of failure (and actually being in budget and time doesn't always mean success). Take the following examples provided by Hugh Woodward, PMP, in Expert Project Management - Project Success: Looking Beyond Traditional Project Metrics:

• Sydney Opera House: With its graceful sails dominating Sydney Harbor, the Sydney Opera House is arguably one of the most recognized buildings in the world. Yet, from a project management perspective, it was a spectacular failure. When construction started in 1959, it was estimated to cost $7 million, and take four years to build. It was finally completed in 1973 for over $100 million.

  [...]

• Project Orion: This massive effort to develop Kodak's new Advantix photographic system was reputedly very well managed from a project management perspective. PMI recognized it as the 1997 International Project of the Year, and Business Week selected the system as one of the best new products of 1996 (Adams, 1998). But Kodak's stock price has fallen 67% since the introduction of the Advantix system, in part because it failed to anticipate the accelerating switch to digital photography.

• Corporate Intranet: Finch describes a project that involved the implementation of a corporate intranet to globalize and improve communications. From a traditional project perspective, it failed to meet its success criteria, but not significantly. It was one month late and believed to have been accomplished with a small budget overrun. But both the project manager and senior management viewed the project as successful. The hardware and software had been installed successfully with a minimum of disruption, thereby providing all staff members with access to the corporate intranet. Following implementation, however, employees made only limited use of the intranet facilities. The main objective of the project was therefore not achieved. In this case, both the project manager and senior management focused on an objective that was too narrow.

  [...]

• Manufacturing Plant Optimization: A paper manufacturing company with five plants across North America decided to increase its manufacturing capacity by embarking on a de-bottlenecking program. A project team was formed to install the necessary equipment, and charged with completing the work in 18 months at a cost of $26 million. But almost immediately, the project team was asked to defer major expenditures until an unrelated cash flow problem was resolved. Rather than stop work completely, the team adopted a strategy of prototyping the technologies on which the de-bottlenecking program was based, and actually developed some cheaper and more effective solutions. Even when the project was authorized to proceed, the team continued this same approach. The project eventually spanned five years, but the resulting capacity increase was three times the initial commitment. Not surprisingly, the company immediately appropriated another $40 million to continue the program.

  [...]

So in these examples, which ones were truly successful? Examples like the 2002 Winter Olympics and the Batu Hijau Copper Concentrator would suggest that these are truly successful because they not only met the traditional project managers' definition of success, but also met the projects sponsors' perception of success.

As we start to look at the examples like Project Orion, the Corporate Intranet and the Laptop Upgrade, we notice that the traditional metrics start to fail. These projects are considered successes in project managers' definition of success, but failed at meeting the sponsors' success criteria. The project Orion example is quite astounding as this project was recognized by PMI (Project Management International) in 1997 as the International project of the year. Yet it did not increase Kodak's revenue, because they did not foresee the adoption of digital cameras.

Most interesting are the examples of the Manufacturing Plant Optimization and the Sydney Opera House. They both failed to meet the traditional project managers' success metrics but were in fact considered successes. This is particularly shocking when you see that the Sydney Opera House had a "cost overrun of 1300%" and a "schedule overrun of 250%".

Once we realize that projects can fail to meet the traditional metrics of success, but still be successful to the stakeholders, this creates a quandary for the project manager. How does one really define success? Is it possible that a "Challenged" project could be canceled that would have met the sponsors' needs? Is it also possible to identify a project that should be canceled that is currently on time, on budget and meeting the defined needs?

What do you think of that conclusion? How do you really define success?

Answer 25

My answer is rather unusual from the rest of this, but quite common around here: killer bugs. I had a project go an extra two weeks (50% extra time) because of one switch in source that I didn't know about until I dug through the source code (there was nothing documented in help or on the web).

Answer 26

I think this thread managed to gather the biggest group of tallented unhappy software developers, engineers, project managers, etc. that it is possible to gather in one place.

I share point of view with most of the posts stuck here and I think they come out of a lot of suffering through seeing co-workers not doing a good job when job (programming) and success is the most important part of our lives.

http://www.clean-code-developer.de/ They have an incredible cause! and their philosophy, if taken ahead, could manage to create a new layer of heroes, as the main stream of developers and IT professionals is so ROT these days.

I'm working on something similar here in Brazil, 'cos I love our profession of bringing software alive both as PM and software developer (.NET) and I can't stand people who face programming as nothing else but they way out to make big money with almost no effort.

... sure I don't consider going overnight in front of the computer geniality. but the few who matter, have done it more than once.