Wouldn't it be closer to:
n * (n - 1) / 2
The above formula is the answer to this middle school Math-team problem:
"You have n people in a room and they all shake hands with everyone else. How many handshakes took place?"
Wouldn't this also apply to the number of people communicating within a software project?
Disclaimer
I haven't read the book (yet), but I've seen the n^2 formula referenced elsewhere.
You are correct. However, while I have not read this book myself, it sounds like they are attempting to give a growth order, not an exact number. n * (n-1) / 2 is a function that grows as O(n^2). See Big-O Notation.
I think the difference is that shaking hands occurs once and counts for both people. Communicating with teammates can be initiated from either person, so you end up counting the paths twice, once for each initiator.
In my personal experience, there are certain people (who I've just decided to call UberCommunicators) that can increase the factor by another order of magnitude simply due to the inherent costs of communicating with them. They tend to be extremely verbose, unable to state a concise point, and generally difficult to keep on task. Getting a useful dialog requires repeated efforts over an extended period of time.
The mythical man month refers to overall algorithmic efficiency, as measured by the behavior near infinity.
n*(n-1)/2 = O(n^2)
If you had read the book, you wouldn't have asked the question. Here's what it actually says:
If there are n workers on a project, there are (n^2-n)/2 interfaces across which there may be communication, and there are potentially almost 2^n teams within which coordination must occur.
For those playing the home game, this is in Chapter 7, under the heading Organization in the Large Programming Project.
So the answer is that you are right, but that's what the book says too.
I think you're overanalysing this. The mythical man month is not a computer book like Learn How to Program Ruby in 21 Days. It's a business book on how teams interact. As such, it treats people as people, not as machines. Which means that approximations and heuristics are the order of the day, not algorithms and precision.
Please (please, please, please) remember to treat your team mates as human beings and not as just another computer. It will make for a more pleasant work environment and a better project.
I didn't remember exactly what the book says, so pulled my copy off the shelf. You may be happy to learn that in chapter 2 - which is also the essay which gave the book its name - Brooks actually does says that the number of communications paths is n(n - 1)/2, matching what you said. So, as others have said, I suspect that the n^2 "quote" is just a simplification along the lines of O(n) notation.