Best practises for organising structure/running "builds" on a large solution

Question

I am making a very large web app (currently at 70 projects and 150k loc but with a lot more to do).

I use FinalBuilder to run build scripts. However, what are the best practises for structuring such a large project? What about build dependencies? What effect does the structure of my projects have on the performance on the code (if any)?

I've seen some previous threads about this but I can't find those. I've seen threads about solutions exceeding 600 projects in the solution, for the sake of clear answers, lets imagine this system will grow to that size (I would like to know how to organise a project bigger than what mine ends up to be, as it would mean I can organise a smaller solution).

If it matters, the system is mostly in .NET 3.5 (C#, LINQ, SQL Server etc) but will use Python/Erlang too.

Thanks

Answer 1

I have only 40 projects (but several millions of loc), and the main best practices we have is:

• identify dependencies between projects
• establish a global list of labels used by all projects wishing to participate to the next release
• make sure that every project willing to publish a label of its own into this global list has made that label from a configuration (list of labels) coming from the global one
• register the "officials builds" (the one potentially to be deployed into production) into a repository.

That way:

• developers works and compile their code directly against the deliveries of the other projects they depends on (as opposed to download the sources of the other projects and rebuild all in local).
  They only have the right deliveries because they know about their dependencies (both immediate and transitive)
• testers can deploy quickly a set of deliveries (from the global list of labels) to perform various tests (non-regression, stress-tests, ...)
• release management can deploy those deliveries (after having a final global build) onto pre-production and production platforms

The idea is to:

• not rebuild the delivery at every steps
• build it only at the development stage (through a common unified building script)
• build it again before release (for pre-production and production platform)
• compile and/or test only against those deliveries (and not against sources downloaded and re-compiled for the occasion: when you have more than a few projects, it is just not practical)

Main best-practice:
If your own project works with the deliveries of the other projects (and not with your local re-build of those other projects), it have good chances to work in the next steps of the software production life-cycle (test, pre-prod, production)

Answer 2

Have you considered using NMaven and making each of the 70 projects a module? That would allow you to control the building, packaging, versioning, and release of individual modules and the parent project as a whole. It would also help you resolve the depedencies between the different modules, external libraries, and even versions and different lifecycle scopes (for example, you only need NUnit during the testing lifecycle, but don't need to package it in the build).

It might help to explain in greater detail what these projects look like and how they depend on each other.

Answer 3

A bit open as a question. Let's start with a basic structure I suggest as a starting point to my customers, inside a branch I have

1. Build Scripts
2. Build Dependencies - things to install on a build machine
3. Libraries - LIB, DLL, ... directly referenced from projects
4. Documentation Sources - help sources
5. Sources
6. Deploy Scripts

then Sources is organized in

1. Admin - admin console and scripts
2. Database - schemas, scripts, initial data
3. Common
4. Web
5. NTServices
6. Services - REST/SOAP services
7. BizTalk - you name things specific to a product