The morning-after syndrome and the weekly build solution

Question

Hi,

I'm working on a small project for the last 7 months and we have been using continuous integration and daily builds since then. It has been quite well for my team.

Today, reading about The Acyclic-Dependencies Principle, Robert C. Martin mentioned the "morning-after syndrome" which happens when you go home after making some stuff working, only to arrive the next morning and find your stuff broken due to a somebody's change in something that you depend on.

To solve this problem, he describes, in his book, The Weekly Build Solution, which works like this: all the developers ignore each other for the first fours days of the week, working on private copies of the code and ignoring integration for a while. Then, on Friday, they integrate all their changes and build the system. Of course, as big as the project, more complex the integration and more time it takes to finish. So, it solves one problem, but creates another one.

So, in big projects, what is the best approach for builds management?

Answer 1

I've used both systems and in practice if the build is easy and fast there's no reason not to make it daily.

It's not hard to try both daily and weekly builds and just use whichever seems to fit best. It may even change over the course of your project. In the earlier stages, less-often builds may be appropriate as larger, coarser, changes and designs are made.

So in this case, there is no "best practice" since it's dependent on your project and circumstances. Try them both and it will be clear if one or the other is preferable.

Answer 2

With all due respect to Robert C. Martin, I'm not sure how up to date he is on Continuous Integration.

With CI, every checkin builds everything affected by the checkin. All the affected unit tests run. If something broke, it will be known quickly.

If someone does a checkin Friday 5pm, then goes home before the build results are known Friday 5:15pm, then maybe they need to stay later - or quit.

Answer 3

I don't know about you... but four days development is a lot of development to have to rework if everything is broken on Friday.

Personally, I'd prefer to break early and break often as I'd prefer to know what is broken as it breaks.

Using a build server should tell you when something is broken in near real time so why you would want to work in isolation for four days and then have a "merge" day is a little beyond me

Answer 4

On our team, which is ~12 developers working on the same product, we:

• have a new build on every check-in
• which is automatically deployed to an automated test server

• and run through a battery of automated QA.

• Everyone gets an email if the compile or the tests fail (with the userID of the offender :) )

We check in CONSTANTLY, as in dozens of times per day, or more. If every checkin is a single unit of work, in the unlikely event of a conflict, it's a breeze (literally seconds) to merge changes.

We also make it a general courtesy not to leave the office until the build initiated by our checkin completes (usually about a 20 minute wait). Not everyone does this all the time, but for the most part it works.

Answer 5

I totally agree John, if you're using CI you shouldn't have the issue, the idea of waiting until Friday, and then wasting everyone's time in a finger pointing bug fest sounds ridiculous.

Having said that I've been on big projects where DB changes were applied by the DBAs on a nightly basis, and that often caused a code break. We addressed that by saying any changes went in at 10:00AM, and that gave a whole day to fix issues, and the devs would know that a break at 10:05 was likely a DB change, needing them to rebuild their local DBs to find the problem.

Answer 6

Having worked on a 50+ engineer team:

1) One engineer gets to be the "build engineer" (permanent or rotating), and it is their explicit responsibility to resolve merge problems (including tracking down people who cause conflicts) from branching, etc. Make sure your engineers are regularly getting the head revision (I suggest daily) so they can see conflicts before they check in.

2) Split the project into different compilation units that can operate purely on well-known function contracts. (Eg, a clear contractual division between a GUI and DB). Create good API's to support this.

3) Have QA give builds a once-over before user deployment. Automated smoke tests if you can.

4) At minimum, daily builds. A week is far too much time to have gone by to discover problems. I suggest using an automated build system that continually pulls and compiles, and maybe even smoke tests too.

5) Breakages (from hard crashes to warnings) take engineering precedence above everything else, no exceptions. Chances are your non-engineer managers will not want this since they do not understand the necessity, so you will have to "work around" them.

6) Keep old builds around as long as you can, to help pinpointing when a particular item broke.

Answer 7

I know one company that does something like this with subversion (depending on the version control system you can do something like it as well):

Each commit to the version control server compiles the code and then runs the unit tests, the commit is cancelled if the compile or the tests fail.

Of course this means you must have tests, but it does stop the commit non-compiling code regardless.

Of course this is problematic if you have a very large code base (on the other hand it encourages modularization which is a good thing).

In the end if people keep breaking the build it is a human issue and you really need to solve it in a human way. Yes you can use technology to enforce things, but at some point the offending people need to be talked to.