How can I avoid pitfalls passing the finished product to the install team?

Question

Anyone out there have any experience with passing off a finished product to the install team?

Our product is installable via RPMs but also requires copying some MySQL data, modifying some config files and running a few development written scripts. Having an install team is great, but it is development that is on-call and after each install we get calls from the customer after hours that we have to support.

We do learn from each incident, but it would be good for our customers, our reputation and my sleep to be a bit more proactive.

Specifically:

• What tools have you used to improve communication/collaboration between the teams?
• What technical solutions have you used?
• What policies have you used?
• Has anyone had any success with writing post-install validation tools?

Edit: I should clarify that I'm not talking about software failures that development and/or QA should have caught. The problem is calls from the customer saying "Option A isn't available all of a sudden" because it wasn't configured to be on, or "I can't log on" because the authentication server is not configured correctly.

Answer 1

Not targeting your specific questions, but once-upon-a-time there was a team of developers and testers deploying a web application to a set of servers for testing and verification.

When it was time for release the customer got the deployable files and deployed it as per specifications ... To a set of servers nothing alike our development servers.

There was a massive influx of bugs, chaos ensued, the customer raged and the poor developers didn't get to sleep.

My tip is one of the most obvious; make sure the development environment matches the production environment to avoid environment specific bugs.

Answer 2

Yes. Some ground rules:

1. Always deliver the product signed and sealed. Use zip (or anything that has checksum), don't deliver individual files or directories.
2. Burn it on a CD and physically hand it over. That way you'll know you have a hard copy (and a backup) that works. You'd be amazed how easy it is to screw up an installation because of CD burners that corrupt files silently.
3. The install team (or QA) should receive exactly what the customer receives, nothing less. Assume they know less about your product than the dumbest customer.
4. Naturally you should always keep a repository of all such deliveries cataloged by version.
5. Print any installation/deployment/user guide that comes with the version and physically hand it over. As paper. Even if the document did not change since last version. I have wasted a lot of time helping QA debug an installation and later realizing they were using the wrong installation guide.

Answer 3

This is basically Assaf's answer with different emphasis. Having been on both sides of deployments, there are two major items to ENSURE a good deployment.

---

1. Few moving parts

This means, if you have an option of giving a few files and having the deployer put them in certain folders on the production environment, or you could pre-place the files in the folder structure, and have the deployer just copy it into the root. Or even easier, a batch file. Or an MSI. If they have to run SQL scripts, then clearly show where they are.

Basically, this step boils down to allowing the developer to create scripts and batch files, and automate as much as humanly (heh) possible. That way, the deployer (who does not know the app as well as you do) isn't expected to mind-read what they are supposed to do with the three files left over. (Duh, you're supposed to place them in folders A, B, D and ZZ)

---

1. DEPLOYMENT GUIDE

That's all in caps because it trumps step one. I am talking about a VERY thorough guide.

It shouldn't say

"move the Map-related files into the Map-App-Data folder."

It should say

"*Move files x,y,z (located in folder X in your deployment package) to the Map-App-Data folder (located D:\AppName\Map-App-Data)*".

Go through the motions of even saying "Remote in to X server, then do y" because you might think it's clear which server the deployer should be on, but for multi-server setups, it's can get pretty sticky as to what should be done where. Given a document this thorough means that anyone can deploy, even someone who you haven't had the chance to train as to what is happening.

---

2.1 Rollback plan

Put the rollback plan right into the deployment guide. If a deployment goes wrong, and they occasionally will, you don't want to leave the server offline until the deployer is able to wake someone up who knows what's going on. It should be there right in front of them. Even if it seems obvious and simple to you, remember that you've just spent the last four weeks engrossed in this project, and this person has spent the last 20 minutes. They simply can't be exptected to know what you don't tell them.

---

2.2 Test the deployment guide

Go through the steps yourself. Or better yet, get a colleague who is NOT on the project to try deploying to UAT with your guide, and you sitting next to them. Anywhere they get it wrong, change the guide. Anywhere the deployment goes wrong (situations you've seen before) add a footnote in the guide that explains why this situation occurs, and how to fix it if possible. It is critically important that your deployment guide not have errors, because when you write the deployment guide, then you are essentially doing to the deployment (because you know how) AND you get the bonus of sleeping through it. But, it also means that any mistakes are on you.

Please add comments for anything I've missed, and I'll toss it in.

Answer 4

• Don't have an "install team". The developers should be responsible for developing a working system, not a pile of pieces that they throw over the wall for some other poor saps to get working.

• Have a fully automated deployment/upgrade process. Deployment should require no manual decision making because someone will one day accidentally make the wrong decision.

• If a deployment fails, fix the automation and re-release. Don't kludge the system into production in-place, because people will one day forget to check the fixes back into the source code repository.

• Test the deployment/upgrade process regularly during development. Preferably on every commit, as part of the continuous integration process.

• Ensure that the test environment is as close to production as possible. Ideally the only difference should be passwords.

• Run environment tests as part of the deployment. I usually implement these in the system itself, so that it self-tests and reports what is wrong with its runtime environment.

• Make it easy to roll back failed deployments.

Answer 5

After some discussion, it was our build team that came up with the idea of using something called Jump Start. We can package our RPMs and any necessary configuration (MySQL commands, changes to httpd.conf, etc...). Once we encounter a problem with an install, we can modify the script; this pretty much guarantees that the same mistake will not be made twice.

I'll update once we actually start using this live.