What are the challenges of transition a team in a corporate atmosphere from a traditional non-iterative, spec list, gantt chart, phase dependent team to a more iterative approach?
Moreover, what was a successful way to gain acceptance with other groups while using a newer development strategy?
You may be interest in the book Fearless Change: Patterns for Introducing New Ideas by Lynn Manns and Linda Rising. It is a compendium of experience in introducing agile methods into organizations.
I'm in the process of trying to do this right now. We've got an on-site Customer Development department and I can tell you they are key in trying to grab buy-in for an iterative development process.
Some great answers on this one here.
If you've already got a track record of delivering projects late and over budget due to large and unmanageable chunks not getting done, that's a good start in convincing the stakeholders of your projects to get the ball of change rolling.
Process can prove itself, but only with the right parties in support of it. Your key is getting other team players to see value in what you're trying to do.
For us, it comes down to approaching things from a customers perspective. We need to constantly come back to the customer to make sure that what we're building is what they envision. We want to streamline the process to stop wasting everyone's time.
Now of course, different parts of Agile work for different organizations and very few companies that actually use Agile processes are doing so in a purist sense.
Through trial and error figure out what works for your business, culture and team. There is nothing that says you can't gradually adopt the overall process and cherry pick the parts that work the best for your business model.
Around here it started with one team who wanted to go ahead and be more Agile using Scrum. This team was the "pilot team" and went for a few sprints (3 months). They were coached by someone from the inside who was already reading and learning about Scrum. Doing a "pilot" instead of a complete switchover helped to gain acceptance from management and refractive team members.
Having a "let's try it" attitude really help to get customers into the process also (internal customers here, but customers none the less).
And to make it stick, you have to take note of the progress made and the advantages it brings to you and your team (it can be better performance, better communication with team members/customers, better results, happier customers, etc.)
In my experience, transitioning the team isn't the problem. It's transitioning management.
What we did:
Explained to management that a plan (which intends to predict the future) is simply fluff, vapor, a list of assumptions without a factual basis.
Planned a list of sprints. Wrote a burndown chart. Forgot to put in summary estimates.
Started executing on the list of sprints.
After the first two or three, management started to realize that the "plan" was just a burndown list, with no "dates", "risks", "assumptions" or anything like a traditional waterfall project plan.
At this point, of course, it's too late. We've already completed one sprint and are most of the way through the second. The horse is out of the barn. The bell was already rung.
So management demands some stuff.
The total budget. We said "Add up the sprints that are important to you. Just draw a random line, anywhere you're happy. That's your budget." No one likes this, because it's too much control. "How can you justify that?" they asked. "Easy. We build in priority order until you cancel the project."
What we did have to add was a tentative duration for each sprints. Ours are of variable size: 2 to 4 weeks. A list of 10 sprints was about 26 weeks -- 6 months. After that, we stopped putting down numbers.
The list of "assumptions". We just refused this. "Write your own". They couldn't think of any on their own. That was that.
The list of "risks". Again, we asked what scared them. If they were bothered by something, then perhaps they should change the priority in the burndown to address that.
A due date. We turned this around and asked them to prioritize the burndown around dates and budgets and risks that were important to them. We didn't much care what order -- it's their call as managers.
After two more sprints, they stopped making "waterfall" requests and started prioritizing and managing the burndown.
Interestingly, they never asked about scope creep. Managers who ask "how do you control scope?" are actively rejecting incremental development. They're trying not to get it.
When managers want to know how Agile methods "prevent" scope creep, they're (a) labeling the learning process as "creep" (which is bad) and (b) resisting the idea that learning leads to scope changes. The only way you even have scope "creep" is when you commit to a specific scope irrespective of any learning that may happen. Agile methods only commit to a next sprint, not a comprehensive scope. If you don't commit to a scope, it can't creep because it doesn't exist.
From my (admittedly limited) experience, make sure that all your programmers are involved with the decision to switch to Agile/Scrum/whatever, and that they're all in favor of it - or at least not going to actively oppose it. I've seen resistance from team members when Agile/Scrum was perceived to be mandated from above without their consent/input. It's hard enough to retrain managers without having to constantly convince your team as well.