Continuous Integration vs. Nightly Builds

Question

Reading this post has left me wondering; are nightly builds ever better for a situation than continuous integration? The consensus of the answers seems to be pretty lopsided in favor of continuous integration, is that evangelism or is there really no reason to use nightly builds when continuous integration is an option?

Answer 1

Yes, if you have a process you want attached to a build, but it is resource heavy. For example, on my team we run JTest during the nightly build. We can't run it during the day because:

1. It requires a lot of resources, which may not be available
2. It takes 4 hours to complete each time

Answer 2

In my professional opinion, the only reason to use nightly builds is when the build process takes so long that it can't complete in a "reasonable" amount of time.

For example, if your build process takes 5 hours to complete there is really no reason to do a build on check in.

Beyond that, there is so much value in knowing as soon as possible when a build fails that it overrides other concerns.

Answer 3

If you're really doing continuous integration with all available tests, nightly builds would be redundant, since the last thing checked in that day would already have been tested.

On the other hand, if your CI regime only involves running a subset of all available tests, for example because some of your tests take a long time to run, then you can use nightlies additionally to run all tests. This'll let you catch many bugs early, and if you can't catch them early, you can at least catch them overnight.

I don't know, though, if that's technically still CI, since you're only doing a "partial" build each time, by ignoring some of the tests.

Answer 4

In our organization, nightly builds and CI builds have two distinct purposes. The CI build is a 'latest code' build in which the unit tests are run against the last check in as you would expect. We also run several code metrics on the CI build.

For nightly builds, however, we only incorporate source code that has been through the peer review process and is deemed ready for testing.

This way, the nightly build always contains build that is 'feature ready' for testing, while the CI build contains features that while functional (to the extent that the unit tests pass) may not be ready to send the to the test group.

The test groups writes new CRs exclusively from one of the nightly builds as opposed to the CI build, although those are also available for informal exploratory type testing.

Answer 5

It depends on the purpose and length of each of your builds. Basically, you should identify what you are trying to learn from CI and decide if it is worth while spending the resources on running multiple builds.

We used continuous integration at my last job for a few different purposes.

First, we used it to make sure that the repository and thus the developers always had a version of the code that compiled. Few things are worse for team members than having to manage another person's broken changes through commenting, uncommenting, reverting and merging, because one person checked in bad code. For this, we had a build that ran instantaneously with no tests or other validation so we knew as soon as possible if the code was safe to update. Builds usually took about ten minutes and the machine was probably running around 50% on a normal workday. No documentation was generated here, just a quiet pass or a loud fail siren.

Second, we wanted to know as soon as possible whether or not any rules were broken. The quicker that you find a broken rule, the easier it is to fix. For this purpose, we had a separate machine that ran a full build and validation of the code. This machine was running 12-14 hours a day continuously on a normal workday. Email status of the build was sent out describing broken unit tests, code compliance, etc.

We stopped there as far as automatically triggered builds go. A nightly build on top of that seemed a bit extreme for us. But I suppose if you wanted to have a snapshot build archived daily, you may want to schedule a third build with the extra steps required for that. Though, we did have another build that wrapped up and archived our QA deployment artifacts for quick and easy deployment, but we only ever manually triggered that one.

Answer 6

If you have a nice robust CI process in place a "nightly" is still useful.

1. As mentioned, a "nightly" build can do exhaustive tests and perhaps some high-level system tests. End-to-end stuff.
2. The concept of a "nightly" build is easily understood by everyone in the organization. If you have trouble communicating CI builds out to other groups (for example, a QA group that doesn't have the same handle on Agile that the Dev group might) a "nightly" is a powerful and simple concept.
3. If your nightly is a separate set of resources, it can be managed separately and used to cut "gold" images with some claims to software integrity. For example, developer writes code, some trusted build system that dev can't touch builds it, QA tests the gold build and signs it. In such a situation, the nightly build functions like a production build system.

Just some thoughts.

Answer 7

I think the other posts cover the common reasons, like having a build process that takes "too long" or having to run only a subset of tests during the CI build. But there's another reason which is political.

In some organizations the official builds are handled by a minimally responsive build/infrastructure/release management/SCM team. In these cases you might put them in charge of the nightly build and then run the CI build out of development. This avoids a fight because their build remains "the official build" and your CI build give you the feedback you need.

Answer 8

We have both continuous integration and nightly builds in place. They serve two different purposes.

Our continuous integration mechanism builds the software and runs unit tests under the continuous integration suite.

Our nightly build tags the source under version control, builds the software, runs the unit tests under nightly build suite. The software built here is then used in various system tests and stress tests.

I think that one of the main differentiator for nightly build is system tests.