Version controlled production environment

Question

My question is, how do I version control the production environment in a good way?

This is our current environment:

• (Internal server) Development - Version controlled source code
• (Customer server) Acceptance test environment
• (Customer server) Staging environment
• (Customer server) Production environment

When we release new functionality for acceptance testing, we make a publish in Visual Studio, zip up the changes and apply them on the Test Server. There we make a backup folder (so that we can revert the changes) and we make a release folder so that we can move these changes to Staging when they're approved.

This is a lot of manual labour creating backup folders, release folders, recreating the directory structure and try to track what functionality that goes into what release. It is teadious and there is always problem with some developer not following the release procedure.

---

In theory I could make a repository for the test environment. (forget source code, this is about the published application) On every release the developer do a commit and supply a comment about the functionality he's releasing.

When functionality should be moved from Test to Staging we export changes made from the last date Staging environment was updated and copy them into the Staging application. There we do a commit that later can be extracted for release to Production environment.

The drawbacks of this is that using subversion will clutter the application with those .svn directories. This may be mended by disallowing access to those directories in the IIS or web.config. Another solution would be using Git on the directory above the root directory of the application. But Git is harder to work with for an unexperienced developer in a windows environment.

Does anyone have experience of this problem? How do you version control your production environment? What if you need to revert a release, do you have a backup folder that you created before the release?

I've discussed this with our developers, and they can't see any problem with using subversion for versioning and backup of the test/staging/production environment. Quite the opposite would they be happy not to create release/backup folders every time they need to release new functionality.

At the same time there is some insecurity about this. None have heard of this before, having the application in a version control system and we are unsure what the drawbacks would be.

If you have experience of a scenario like this, I would be happy to hear about it.

Mikael Lundin

Answer 1

I been using mercurial (Hg) as a production version control system. (Not the code, but the actually deployed binaries). That works quite well. Subversion isn't quite right to use in my scenario because I do want to be able to sync production changes (such as config files) back to testing and maybe even development. subversion is harder to sync/merge between different repositories ( a lot of manually coping). And with hg tag and hg update there is a breeze to revert to a stable (or unstable) tag.

Oh, and I totaly agree, you should use a buildserver (cruisecontrol.net) or something and keep all your things there as well. I my scenario that isn't enough because the config of a customers production system is, well, custom specific, and even with extensive testing and what not there can be some config that no longer do the same thing between two releases (or the incoming data has significantly changed)

Answer 2

Another way to do things would be to use a build server. Every time you check in code the build server builds and packages up the new build inside the development environment. Then checks in the deployable package with your code.

You can then deploy the packaged build to the other environments. This way you don't have to back up the versions there because you already have all the built versions in your main repository. If you make sure deployment is fully automated and can be done with one command (powershell?) there's no need to keep backups on the servers anymore.

This is actually better and more maintainable than the solution you propose. Because every version of the build is kept alongside the code you can always find a complete development environment for any build package. If you version-control your servers separately and you find a bug somewhere it can be hard to find the version of the code that causes the bug.

Answer 3

Thank you for your answers and suggestions.

I see that I need to rethink the way we handle releases to these environments. Having a build server with all the releases ever made could solve a part of the problem. I will get the commit comments on the source code, and much better control of what feature goes into what build. I still need to keep track on what build is in what environment to be able to revert a release if something went wrong.

@Jonke I will take a look at Mercurial. Maybe a version controlled production environment is overkill if you have a build system where each revision has version controlled source code. But I would still like to have a fast way to revert changes to a previous revision if anything goes wrong during deployment of new functionality. I also like the way you get version controlled configuration files on the server, since production environment always have different configuration from the development environment.

Maybe I'm looking for the silver bullet :)

Answer 4

We use subversion as a way to deploy things to our different servers(prod, demo, dev, etc.) and have not run into any difficulties. The only things to watch out for are database differences and config file differences. The config files can be templatized, so as not to overwrite the files in each different deployment, but then you don't get those versioned for the changes to that specific platform.

With this sort of a system, you can make use of subversion's tags to update/roll back to specific deployment versions easily.