Requirement Clarity - How much is too much?

Question

It’s been only recently that I have been responsible for requirement gathering and I am quite overwhelmed by my inability to capture ALL requirements. At the end of a release requirements slip and I am, to say the least quite annoyed with me.

In the upcoming release I seem to be going crazy. I have one too many questions that I would like to ask the client as I don’t want to MISS ANYTHING.

Now the client would very much like something so accurate, but I have this feeling he might not want to be nagged day in and day out or bothered with every minute detail. (The only problem being, once delivered this detail that seemed so minute is not so minute anymore. Do I make any sense?)

Am I allowed to / supposed to ask the client ALL doubts that could lead to possible requirement gaps? Or should I just do what seems most rational to ME - hope things go fine, and if not just deal with it after the release?

---

All answers below (and all discussions I have had with colleagues) seem to recommend Agile / TDD / Prototypes / Iterative development. I am very pro some of these ideas and would love to follow them given a choice. I however don’t seem to have one at the moment, as the project follows (for over 2 years) a very “traditional” approach to software development.

From what I gather, the only way to move forward is to look for resources that would help me move (an existing project) from a traditional to an agile environment. Any leads? (Is this a terribly daunting task or is it just my imagination?)

Answer 1

You really should take a more iterative approach. Get the basic details, and then do up prototypes and have the customer tell you what they think about it, and give more detailed requirements as the project becomes more complete. Even if you do get all requirements up front, the requirements will likely change after they actually see the project in action.

Answer 2

The inability you feel is a law of nature: A brain with a limited size cannot contain all the information in the universe, only a very tiny subset.

Look back in your life: Did you know everything you know today from the beginning? No.

For any program you ever wrote: Did you know everything you know today from the beginning? No.

You can't. Forget about the impossible and look at what you can do. Anything you do has a learning curve. That's true for you and your customer. No one knows everything in advance, it's just not possible in a universe with a distinct direction of time flow.

So instead of writing a documentation which covers everything, write a program which covers everything and start with the question: "How would you know that my program does the right thing?"

That's usually very easy to answer for the customer. Then write a test which checks that your program does the right thing in this case. After that, write the part of the program which makes the test work.

This way, your customer can answer the questions she knows. You can concentrate and what you know and when you have made a test green, you can forget about it because the testing framework of your IDE will take care of that.

When you have written tests for all the answers your customer gave you, you're done.

If you need details, search this site for Agile Development, TDD (Test Driven Development) and Scrum. Note that there is no "one size fits all" strategy and there will never be. So if someone tells you "you must do everything or you'll fail", you know you'll fail because that never works. Chose a strategy which you can adjust for every project you do.

Answer 3

Trying to know everything upfront leads to "analysis paralysis." You simply can't know all of the requirements because the customer doesn't know all of the them. They can't know all of them, because some of them are dependent on how you develop the software and because, until they see the software, they really don't know what it is that they want.

I would say that you need to know the "big idea" upfront. You should have enough information to understand the customer's basic goals and needs. To this end I would identify the main actors in the system collect what I call story titles based on how those actors will interact with the system. Have the customer prioritize these and pick just enough story titles to for about 2 weeks to 1 months worth of work; at least enough to have something workable at the end of that time.

At this point give some thought to your basic architecture - basically a sketch of how you think it will be built, not a detailed drawing. Do some deep thinking about the stories that go with the titles in your first iteration, then implement, preferably in a test-first manner. Release the software to the customer to have them use it -- in production if at all possible. Collect their feedback as story titles.

Shortly before your iteration is done, sit down with the customer and have them reprioritize the titles based on their experience with the current version of the software they have, their feedback (new stories), and the existing backlog of stories. Choose enough stories for the next iteration. When the current iteration is done and released, start detailed planning and implementation of the customer's next highest priorities.

Repeat this process until the customer is happy with the system you've delivered and the stories in the back log are not worth the cost of developing them (from the customer's perspective).

Answer 4

What you might be missing is a defined, proven way of requirement modeling.

Neither the client, Nor the consultant will be fully aware about the requirements, right from the beginning. So, you need to follow an iterative process. You need to work with the client, to understand the requirements, and at times, to make him understand what he need.

A good process to follow in such cases is the Scrum process. In Scrum, you interact with the stake holders frequently, and you improve you product through various iterations. So, you can scope your requirements for various iterations, and each iteration ideally should have deliverables.

Everyone knows that you can't build Rome in a day, and any customer will understand this if you communicate this in the right manner. Scrum is an Agile process framework that allows organizations to continuously direct the project toward early delivery of real business value through the frequent and regular delivery of high quality software. In this way, requirement gaps can be caught as early as possible. Of course, I'm not going to detail the entire scrum process - but wanted to hight light that, might be what you are missing is a defined way of collecting requirements.

Hope this helps.

Note - If you are working in Microsoft space, you might consider using Scrum for team system here. http://scrumforteamsystem.com/en/default.aspx

Answer 5

I never worked on a project where there were not requirements discovered during development. If you have a solid foundation of requirments, that is better than most projects.

Answer 6

Gathering all the requirements up-front doesn't work for several reasons, the most important of which is that software development is a wicked problem. To be more precise, it's non-linear.

The end-users have a process that needs to be automated. But as soon as you start to automate that process with software, the users react with that software in unpredictable and messy ways. They will use your software to re-work their process, and this will inevitably lead to new requirements. So you will need to change your software to deal with this, and then go round the whole loop again. Continue until exhausted :-)

There isn't really any alternative to iterating this loop several times. which is why agile software techniques are achieving so much attention. In your situation, I would sit down with the team, outline the problems with your current software development process, and then suggest how delivering software in a more iterative fashion might help. Do not suggest any specific agile methodology, but instead focus on delivering to the end-users more frequently.

At the same time, you need to keep a close watch on scope creep. Many users will sneak-in new requirements, some of which may be very costly. You need to triage and "scrub" those new requirements to make sure that they're essential.

EDIT: You edited your question to ask for tips on moving to agile development. In my experience:

• Find somebody in the business who will champion the agile process. When you have good business buy-in, the IT team will follow.
• Also talk to the other project stakeholders such as the end-users and the project manager. Don't do this in an evangelistic manner by hyping agile, but more in a experimental manner.
• Change doesn't happen without a pain point. So find the project pain points, and see how agile development can help with these.
• Don't chose any specific agile methodology, but rather focus on iterating and delivering more often. I find that taking the total project estimate (say 25 weeks) and then square-rooting that number (5 weeks) works. So with those numbers, the team delivers something useful to the customers approximately every 5 weeks.
• Don't under-estimate the cultural changes involved. For some organizations having the development team working closely with the users of the system is new and disturbing. Users have a habit of interrupting development nirvana with nasty exceptions to business processes. Likewise, many business folks equate working with the development team as having to join their mechanic in the muck and oil of a garage when getting their car repaired. Have your arguments prepared beforehand to address these points.

Answer 7

From your comments on the answers given, it sounds as if the environment you are working in is broken. Your clients do not want a development process that works, rather they want something that conforms to their ideas of what development is. I had similar experiences working for the phone companies (seven large clients, hundreds of millions of dollars floating around). In a situation like this, in order for you not to make yourself crazy, you must do two things:

• Find out what game you must play to satisfy clients, management, or whoever else is requiring you to work in this unreasonable way. Produce something called a requirements document whose only purpose is to make the clients happy. If actual requirements are elicited, we're all surprised and pleased. This is a tax on your time, or in business terms, gathering requirements is a cost center, not a profit center.

• Operating in "skunk works" mode, do your best to identify a reasonable set of requirements and meet them. Develop agilely. Iterate. If they will let you talk to the real customers, show prototypes.

Avoid the false dichotomy posed in your question. You have more choices than gather all requirements up front and do what seems right to you until release. With luck you can walk the tightrope:

1. Gather enough requirements to make it look like you're gathering requirements.
2. Do something small that seems right to you.
3. Show it to the client, ask if it seems right, seems to meet requirements. Perhaps you can call this "gathering more requirements". The important thing is that the client see the prototype.
4. Repeat several times before release.

If you can't do this with a real prototype than build a paper prototype as described in Jakob Nielsen's book on Usability Engineering.

Good luck!