How big is too big for XP/SCRUM?

Question

In the earliest stages of planning the development of a new system, which development model to follow seems paramount. I've always held onto the belief that a classic waterfall (or hybrid waterfall/iterative prototyping) is the best approach for medium to large projects. It seems that once a project gets to be a certain size, the Agile/XP/Scrum paradigms can't account for complex requirements, a large team, the complexities between multiple sub-systems, the need for documentation, personnel changes, etc, etc.

What's the limit of such agile methodologies in terms of system size, team size, LOC, etc?

Answer 1

I don't think there is a boundary, after all the ideas of scrum came out of car manufactoring and that's pretty big in terms of people. The thing with big projects is, that you need to start with a small team and grow it over time. Keep separate teams that interact via Scrum of Scrums and it will scale, if the people are willing to collaborate it will work. It's like always in our business: divide and conquer. Break the big hard problem into smaller manageble chunks.

Answer 2

Within a team the communication channels are proportional to (N * N-1) / 2 as an upper bound, so could loosely be viewed as O(N^2). The decentralised nature of agile teams means that there is no central point of reference and the communication will grow closer to the upper bound than if there was such a point of reference.

Where you have a written specification and a more formal structure (see Painless Functional Specification for a discussion of spec documents) the communication is closer to a hub-and-spoke model, which has closer to O(N) channels (for N staff on the project). Most of the rule-of-thumb commentary I've seen puts the sweet spot for Agile teams at 6 or less and the upper bound at around 10, although your mileage may vary.

In the PFS articles Joel (yes, that Joel) discusses the role of a Programme Manager, whose role is to develop and own the specification. The Painless Functional Specifications series goes into this in quite a bit of detail and is also quite accessible to non-technical management - I've referred quite a few people to this article.

Answer 3

Picture Scrum/XP as a series of mini-Waterfalls. Initially, you want to do an upfront effort to get a good, well-defined backlog. Not necessarily the whole system, I'd argue that once you get one or two sprints worth of product backlog items, it's time to start sprinting. Concurrently with the sprint, you should be creating additional PBIs (and reprioritizing them appropriately).

The idea is that you can get business value delivered before the system is FULLY defined.

Answer 4

Scrum can be scaled using "Scrum of Scrums".

From the Scrum alliance comes this advice on conducting Scrum of Scrums meetings:

The scrum of scrums meeting is an important technique in scaling Scrum to large project teams. These meetings allow clusters of teams to discuss their work, focusing especially on areas of overlap and integration.

The book Agile and Iterative Development also discuss this issue.

Answer 5

Agile scales fine. It is not a rocket science. In fact it is all about modularity. Software development is a CAS (Complex Adaptive System) and, as almost any CAS, it has modules to rule the complexity better. Scrum of Scrums is one of the possible modular approach for development process scaling. Functional divisions (Developers, QA, etc) is an another modular approach. The worst case is when you do not have modules at all in a large project.

Depending on a project nature, team may decide what modules will work for the project. General pattern is to form several teams that work on some low cohesion modules. Each team should be quite autonomous, but interaction with another teams should be good.

The analogy from CAS is a human body for example. We have organs like heart and liver. They are separate modules (teams of cells :) that interacts via nervous system/blood/etc.

Answer 6

Have a look at this blog post by Bernie Thompson.

It outlines a lot of the issues and trade-offs he ran into when scaling up Scrum / XP at Microsoft, and has some very thoughtful and interesting solutions.

There are other posts on the same blog that also deal with these issues of scale that concern you - IMO it's a gold-mine of ideas on "agile for grown-ups".

Answer 7

Scaling scrum or any agile approach is dependent on your environment.

If you have multiple projects with multiple teams, scaling is simply sharing best practices across teams. As soon as you start requiring integration between systems/projects, be wary. Tighter integration between teams is preferable at that point.

If you have one large project (I had a team of 45 at one point), there are different approaches to scaling. We chose to keep one team with multiple standups - developer standup separate from BA/QA standup. The iteration manager attended both and at least one from each side attended the other. We had one card wall, but it included pre-iteration stuff (stories in process of analysis, production bugs to chase) and post-iteration stuff (release/deployment work).

I've also been a part of one very large project with many scrum teams (~20 teams - some distributed - ranging from 10-20 members each). Each had separate standups, and there was a scrum-of-scrums and even a scrum-of-scrum-of-scrums. I think we made a mistake by segmenting the teams by functional area rather than workflows. Our segmentation created silos of code ownership with onerous integration management issues between teams.

In sum, it's not just about size for scaling... it's also about the content of the project. Feel free to share more specifics about your environment to hear more specific approaches to addressing scale in your environment.