It happens because it has to. You can only get away with that when you are a very small (less than 10-20 person or so on a single project) development team. If your project grows and your team grows, either your practices will scale up or you will fail.
The change you are asking about is essentially the transition from hacking to programming and finally Software Engineering.
With Software Engineering you recognize that not everyone in the team is perfect. You review code, you ensure that others test, you cross-check each other.
You start to see the need for an architect that can digest the customers desires and translate them into a design document. This can easily eat a month of time before anyone else is even added to the project (but over the course of development time can save months or even years!). He makes sure that everything makes sense and will fit together reasonably well.
You have design documents, generally UML based so that different parts of the team can understand integration points . You recognize that anything that has been done, you may need to re-do without the people who did it, so you document it.
Your quality process gets much more strict and they start enforcing rules like you only check in changes that address specific bugs during testing.
Testing, refactoring, etc. are obviously key and are re-enforced by peer and team review.
I'm not saying this type of stuff is always necessary, obviously it's not, but you asked why your code base is looking crufty, and this is often the solution to that problem.
Usually this happens after a GIANT project that completely fails because the codebase sucks so bad. Then they fire anyone who can't dodge the blame, hire some managers who hopefully have some experience with larger projects and (if they don't run out of money) restart from ground zero.
At least that's my experience. YMMV