How to improve our build and deployment process?

Question

Hello all!

Our build/deploy process is very tedious, sufficiently manual and error-prone. Could you give proposals for improvement?

So let me describe our deployment strategy and build process. We are developing system called Application Server (AS for short). It is essentially servlet-based web application hosted on JBoss Web server. AS can be installed in two "environments". Each environment is a directory with webapp's code. This directory is placed on network storage. Storage is mounted to several production servers where JBoss instances are installed. Directory is linked to JBoss' webapps directory. Thus all JBoss instances use the same code for environment. Configuration of JBoss is separate from environment and updated on per instance basis.

So we have two types of patches: webapp patches (for different environments) and configuration patches (for per instance configuration)

Patch is an executable file. In fact it is bash script with embedded binary rpm package. Installation is pretty straight-forward: you just execute file and optionally answer some questions. Important point is that the patch is not a system as a whole - it contains only some classes with fixes and/or scripts that modify configuration files. Classes are copied into WEB-INF/classes (AS is deployed as exploded directory).

The way we build those pathes is:

1. We take some previous patch files and copy them.
2. We modify content of patch. The most important part of it is RPM spec. There we change name of patch, change its prerequisite rpm packages and write down actual bash commands for backing up, copying and modifying files. This is one of the most annoying parts because we not always can get actual change-set. That is especially true for new complex features which are spanned among multiple change requests and commits. Also, writing those commands for change-set is tedious and error-prone.
3. For webapp patches we also modify spec for other environment. Usually they are identical excepting rpm package name.
4. We put all rpm related files to VCS
5. We modify build.xml by adding a couple of targets for building new patch. Modification is done by copypasting and editing.
6. We modify CruiseControl's config by copypasting project and changing ant targets in it
7. At last, we build a system

Also, I'm interested in any references on patch preparation and deployment practices, preferably for Java applications. I haven't succeed googling that.

Answer 1

The place I work had similar problems, but perhaps not as complex.

We responded by eliminating the concept of patch altogether. We stopped patching, and started simply installing the whole app (even if we do a just a small change).

We now have Cruise Control build complete install kits that happen to contain the build timestamp in the install-kit name. This is a Cruise Control build artifact.

Cruise Control autoinstalls them on a test server, and runs some automated smoke tests. We then run manual tests on the test server. Then we install the artifact on a staging, then production server.

Getting rid of patching caused some people to splutter, "isn't that wasteful if you're just changing a couple of things?" and "why would you overwrite all the software just to patch something?"

But the truth is that good source control, automated install-kit building, and one-step installation has saved us tons of time. It does take a few seconds longer to install, but we can do it far more repeatedly and with less developer labor.

Answer 2

We are also trying to move towards patch / hotfix releases as well as releasing full installation packages based on RPM.

I must say I also agree that in most cases this effort is a waste and I would also go for releasing the full installation package assuming that you already have an established build pipeline.

In our case we have a multi-module delivery each packaged as an RPM and wrapped in an ISO for delivery. We aim to keep our build pipe-line mostly unchanged for releasing hotfixes. Therefore, we instead focus to make our RPMs more fine-grained (smaller - more targeted RPMs) so that we can only release those RPMs that contain changed artifacts for a certain hotfix / patch.

This way the full release RPMs and patch RPMs are the same, the only difference is the number of RPMs you package in your delivery ISO.

I have another question addressing this issue, you may take a look at there:

hotfix-patch-build-delivery-approach