Continuous Integration Advice?

Question

I'm working on setting up a Continuous Integration server (using Integrity) for my Rails app, and I'd like advice:

• Do most folks set up CI to build and test their app on every push to their central SCM repository, or only when pushing to their staging branch?
• I'll use the CI server to automatically run flay, flog, reek, and rcov -- are there any other testing or code-coverage tools I should run?
• I'm planning to deploy my app on Slicehost. Should I set up the CI server on a separate Slicehost slice that's set up to be identical (wrt installed gems, libraries, etc.) to my production slice?
• If I do use a separate slice for CI, is there any harm in using the CI slice for a staging server as well?

kind regards,

Jacob

Answer 1

It sounds like you may be looking to use the CI server for two purposes:

1) Continuous Integration - This would probably be the pushes to the central SCM repo part. Use your CI tool to validate that the area where developers integrate their work is functioning. This will provide the development team rapid feedback. You should run as much testing as you can get done very quickly here.

2) Build Management - Against your staging branch, builds are probably done so they can be more rigorously tested and managed for further deployments. Here, I'd run all the tests you have setup (and other posters suggest) and probably track these builds carefully.

If that's accurate, I would definitely build off both branches, but your collection of tests and how you maintain your build results for each branch may be different. Then again, if you are able to keep your mainline pretty clean, what does the separate staging branch get you?

Answer 2

Testing & Code coverage tools - you want to include as many of these in your suite as make sense without bogging down your build cycles. e.g. if you add too many and your build times go beyond a few minutes, then it won't be productive to continue to build after every commit.

Build after every commit, IMO, for both your central and staging locations. You may get double-coverage here, but if you have the bandwidth and CPU for it, why not?

I agree with using a dual-slice setup, and yes one of the reasons is that it allows you to use the second slice as a staging/final testing area. Plus then if you deploy from your staging slice to your production slice, you get to take advantage of their local network while transmitting information between slices. It's much faster to do it that way.

Hope that helps :-)

Answer 3

If you have a more formal release process, and a separate test team which can't test as fast as developers check in, you might set up two builds. (1) One very fast, light-weight process with no permanent artifacts or storage, where every build overwrites the previous once complete. This runs constantly against each integration, to tell developers what just broke. These internal-only builds are not controlled by release managers beyond deciding when would be a good time for build type (2), which is a less-frequent build which runs all tests (including manual ones), creates complete artifacts, and stores metadata to number and reproduce the build.

Answer 4

We use Slicehost for deployment and testing. For development we have three slices arranged as follows:

Dev

Hosts a mercurial repository, continuously updated by developers (usually just merging their branch to the main branch). Tests are not triggered on check-in but scheduled hourly. This slice has a MySQL installation with a test database.

Test

Hosts a mercurial repository, updated only by dev manager. Used for acceptance testing but also has nightly test runs. This slice has a MyDQL installation with a test database.

Deploy

After a successful QA, we run a capistrano task which pushed the updates from Test to the Deploy slice. This slice can be considered a "setup.exe" - customer slices should initialise smoothly from a backup image of this slice. Testing on this slice really revolves around ensuring that this can happen.

New customer setup

Slicehost allows you to save three copies of a slice so we can go back several incarnations of the system.

When a new customer needs to be setup we initialise their new slice from the latest version of Deploy, add a cert for HTTPS and register the new slice in the DNS.

We also store things like /etc/nginx/nginx.conf and /etc/mongrel_cluster under version control.

Chris

Answer 5

Do most folks set up CI to build and test their app on every push to their central SCM repository, or only when pushing to their staging branch?

Depends on the team size and velocity of the project. The more people I have working on diverse branches of the code, the more often and vigorous I want the CI to run. I would recommend you start with as much CI as you can muster and gradually back it off as you see fit.

I'll use the CI server to automatically run flay, flog, reek, and rcov -- are there any other testing or code-coverage tools I should run?

Everything covered by metric-fu is a good start. If your team has a tech writer and/or documentation is part of your delivery, you can throw rdoc in there as well.

I'm planning to deploy my app on Slicehost. Should I set up the CI server on a separate Slicehost slice that's set up to be identical (wrt installed gems, libraries, etc.) to my production slice?

If you can afford it, yes. Usually small teams and fresh start-ups can't afford to have a dedicated server for every task, but I'm a huge proponent of isolation. Regarding the identical set-up, vendor everything that you can; setting up a fresh server should be quick, simple, and automated.

If I do use a separate slice for CI, is there any harm in using the CI slice for a staging server as well?

The only harm is cross-contamination between shared resources, e.g. gem versions, database resources, etc. If you're methodical you'll probably be OK, but if you can afford separate servers per environment I'd lean on the side of better safe than sorry.

Answer 6

I recommend you read this.