Predictive vs Reactive software design

Question

I know that for me I first got started following the waterfall method of project management and along with that I went with the predictive approach to software design. In this I mean we had huge packets of documentation, UML, database schemas, data dictionaries, workflows, activity diagrams, etc.

Having worked in software for over 10 years now I find it to be much more realistic to approach software design from a Reactive approach. I frequently follow a scrum approach to project management and with that very little heavy documentation is ever generated. We have very little workflow specification (though they still have there use). This is a much more dynamic approach to software creation. Of course along with it comes frequent refactoring as time goes on as we find out new features over time that had we planned for up front would have changed things dramatically.

The big difference for us is that the first approach takes longer, seems to fail more frequently in a software construction world, and isn't nearly as flexible. The second approach provides more flexibility, makes us aware of failure faster (so we can course correct faster), and provides some form of functionality at the end of every iteration.

Knowing both sides from experience, I still find many people that LOVE the waterfall approach over the agile approach for software development. I don't get it.

question: Why would someone use waterfall over some form of agile with all of the research backing agile? What are strong arguments for using waterfall over agile?

Answer 1

My boss tells me to.

I suspect many people have no choice and old bosses don't learn new tricks.

Answer 2

When I started programming (with COBOL no less), waterfall was the "new" approach. Today, I'd tell you that I use a waterfallish agile methodology. For larger systems, I find a waterfall type start works best. Not creating huge documents (that's a waste of time IMO) but rather to take some steps like creating a UI prototype and/or use cases to get our heads around the business problem at hand. Once we are comfortable we have the problem scoped and we have a solid understanding, we move into an agile development mode.

To answer your question though, I think the big reason waterfall sticks around is many people don't like change. It's scary to change and moving from waterfall to agile is a big change.

Answer 3

If you exactly know the requirements that never chagen, if you know how long each step will take and if you know all the resources are avaliable at the time needed you can do waterfall and it will work. But in deed these kind of projects are quite rare and I think I will never be part of it.

Answer 4

When designing systems to be used by end users, agile often works well because the requirements are likely to be incorrect and a large part of the process is getting feedback from users in the form of a usable version.

However, when creating software that interfaces with other software often the requirements can be worked out in very clearly. In this case it is often more productive to ensure that you have a very clear and accurate specification, unit tests In this model you can also generate fairly good work estimates and there would be a great deal more cost to use the agile model.

Answer 5

I think that part of the reason why people still often cling to waterfall is that it gives the illusion of control. In a waterfall, you can do enough up front work to put together a beautiful schedule that nicely addresses every contingency that you can think of, and then give a detailed roadmap for the future to anyone on the business side who asks when feature X will be available.

The problem is that you can almost never follow that plan to the letter, and you are almost always late/dropping features. However from the upfront, it looks very controlled and manageable.

I'm a big Agile fan, but what I've always struggled with is the long range roadmap/forecasting that is often asked for by the sales and marketing folks. I think that the waterfall's illusion of certainty is very comforting to managers and business folks.

Answer 6

Not taking sides, but pretty much any research would be unscientific at best.

You say (emphasis is mine)

question: Why would someone use waterfall over some form of agile with all of the research backing agile? What are strong arguments for using waterfall over agile?

but don't link to any studies.

It's one of those things that are known to be extremely difficult to actually test. You can't have two identical teams work on the same project at the same time, because there's no such thing as two identical teams. You can't have the same team complete the same task twice in a row using two different methodologies without the first pass tainting the second. I've never heard of anyone designing an experimental (or even statistical) study that can convincingly argue for any software development methodology. I'd be interested to see one though, if you have a link.

Short of real evidence, it boils down to personal preference. What are the strong arguments for chocolate over vanilla?

Answer 7

I'll play devil's advocate and state that agile is flawed is nearly as many ways as the waterfall method is. I'm not one of those that love the waterfall method, but I don't love agile either.

My experience with agile hasn't been very positive. To be fair, I used it in a corporate environment, which paid lip service to "agile" while still expecting our manager to produce long term milestones and deliverables upfront.

However, I found that agile (scrum in particular) methodologies often disguise major problems with design. While waterfall gives managers the illusion of control, agile seems to do the same for development teams. I've seen teams where bringing up any issue that aren't in the current sprint/iteraton is frowned upon, with the expectation that it'll be handled "in time". It only requires a few major design decisions to be ignored for the project to go belly up in future, while current iterations go smoothly and project looks to be on track.

You can argue that the team's at fault for not understanding the spirit of agile, but I'd like to see better methodologies that incorporate the best parts of agile.

Answer 8

The title says it all. (Actually: proactive vs reactive). Why chose the reactive way and give up control unless you don't have to? Waterfall is not the only alternative, you can have any kind of development process what you refine when you like. Control is the key.

It's a spectrum btw, the waterfall on one end and the totally reactive, zero documentation methods on the other end. If you work in the consultant industry for powerful (and usually indecisive) clients, you have to resort to reactive methods. If you develop shrinkwrap software you can plan ahead and manage knowledge. Some projects also require tons of specifications and rules, where the code and fix approach just don't cut it. For me software engineering is primarily about knowledge management and design, coding comes second.

P.s. there is no such a thing as agile and fixed price. Not in the classical way they usually sell the method. See http://martinfowler.com/bliki/FixedPrice.html

Answer 9

retroactive behaviour

Answer 10

If you have a team of a few dozen people that have over the course of a decade, refined the waterfall strategy to the point that it works well for them, who are you to come in and say, "You're doing it wrong..."? Really, if it is working for them, why change things? Yes this is merely flipping the question around but I think it may be a valid point.

Answer 11

One of the premises of (at least) XP is that change is cheap. The waterfall model was built on the principles that change, any change, is costly. The assumption in the waterfall model is that once software has been written, changing it is more expensive than investing the time up front to come to a "complete" understanding of the problem.

Experience seems to indicate that it is very hard to come to a complete understanding of the problem and that if some precautions are taken (e.g. Unit Testing) change can become a lot cheaper. Therefore if you encounter a problem where some of the agile premises don't hold true other approaches might become feasible again. In between Waterfall and Agile there is at least Spiral development which is - sort of - what we practice.

Answer 12

In my team we've found that with maintenance projects (which is the bulk of what we do) where we're tweaking or replacing like with like there isn't always as much need for user input beyond perhaps some UI prototypes.

In that case, particularly given that there are commercial deals involved the waterfall approach at a macro level can fit well. Even then we still like incremental / agile approaches at the implementation level.

It is worth noting that most of our clients are large lumbering organisations in love with their paperwork, so that adds even more impetus for us to at least appear traditional to them.

Answer 13

The documentation generated during the waterfall process allows for a lot of CYA. You can point fingers when a project goes off the rails. Very few executives are going to be OK with "oh well, I guess that project got away from us! Well, at least we found out early, no harm no foul!"

Also, design docs can automatically generate test plans, which is useful for QA.

Answer 14

You need to be preditive enough to deliver the goods. You need to be reactive enough to deal with the issues.

I was once stuck with six months to complete a project estimated to take a year, and based on past experience experience would take two. So I spent three months researching methodolgies. We finished on time (in three months), using the appropriate parts of a waterfall process.

A few points that made the methodoly work: - Create an use standards, update them when needed. - Build libraries: Do it once, do it well, fix it without breaking existing code. - Do just enough documentation. - Version control everything you can. - Break things down; a method should either manage work or do work. - Increase cohesion, decrease coupling, reuse. - Buy or build the tools you need. - Track your issues and progress.

Another project I was breifly involved was a six month project. I didn't get involved until a year and a half after it started. The development lead had been hired at an extreme markup as he was leaving a career with a pension plan. At the start of the project he asked the project manager, "Do you want me to do it right or be reactive?" Can you guess the answer? The week I was involved same feature was implemented on Monday, Wednesday, and Friday. Guess what happened Tuesday and Thursday?

The strength in Agile is its emphasis on just enough, just in time. The strength in the waterfall methodoly is that it covers all the things you need to think about. I've yet to work on a project that did or should have done all the steps. I have worked on many projects that did steps which should have been done on a corporate basis.

Answer 15

It's pretty common when bidding for a contract that one of the iron-clad conditions is that you follow their "process" which on inspection is waterfall.