What are the best practices for versioning with a data driven web app and many devs?

Question

I've read many answers to similar questions, but still didn't get to the answer that I was looking for.

We've got a group of about 12 devs and business analysts working on one app. It's an enormous application, I'd guess about 1000+ pages in a mix of ASP and ASP.NET.

What I'm wondering is how the pros manage versioning of a large app like this one? Especially how to manage deployments, database changes and source control. Do they build the source control procedures in such a way that the app can be rolled back to a stable point at any time? How does the database fit in to that? Are all database schema changes and procs etc stored in source control?

I think my ideal solution here is to be able to re-hydrate the entire app from scratch to a specific version, including data. Is that overkill?

Update: my first guess at the size of the app was way off. I did an actual count and came up with a bigger number. 90% of those pages are frozen for development or unused.

Answer 1

Perforce has a great white paper addressing some of the issues you raise. From their white paper, that talks about web pages, you can extend the concept to stored procedures and scripts for generating/modifying tables.

As for being able to rehydrate from scratch, that really depends on your disaster recovery plans, as well as, how you setup and configure your development, and integration testing environments; and how much downtime you are willing to accept. I don't deal with these issues day to day, so perhaps people who have a lot more real world experience, specially people on the Operations side of things can impart their wisdom.

Answer 2

120 pages is "enormous"? Wow...

Versioning an app like this is quite straightforward. You have a production rollout process that includes tagging every release in the revision control system before it goes out, and only installing tagged releases. You back up the database at the time of the upgrade, before changing the schema, and store it somewhere with a descriptive name that allows you to link the code and DB backup. If you need to roll back, you just install the old code and restore the backup.

If you need to rollback, but with the current data, well, that's a harder problem, and is more about how your database is structured and how it's structure is evolved, but Rails migrations are an interesting study in how it can be done.

Beyond that, your question is pretty huge, and touches on a lot of areas. It might be best to contract someone who has dealt with all these things before and put you on the right path.

Answer 3

This is really quite a normal, and moderately-sized application. You just need to follow good source control processes, use proper unit testing, continuous integration, etc. Many, many, development organizations have solved this problem.

With all due respect, the fact that you consider 120 pages to be enormous, and the fact that you feel this is a big deal, both indicate to me that you should consider stopping development Right Now, and improving your development process.

I'm concerned that if you don't do that now, you'll learn later why you should have done so.

Answer 4

For something this small, rather then worry about being able to roll way back in time, what you are really after is the ability to roll back the most current release to the last one. That way if you break the build, you can roll back to the build prior. How?

1) Your web server reads from /www/working 2) Push your changes into /www/new 3) Move /www/current to /www/old 4) Move /www/new to /www/current

If /www/current blows up, just move /www/old back in its place.

The only catch is rolling back the database schema isn't part of this. Rolling back schema changes is difficult, if not impossible. All you can really do is restore the old version from backup and loose any activity between the live database and what was on backup. This makes sense if you think about it--if you add three new tables and somehow refactor another table into a better schema, how can you roll back the schema change after that? You can't--you just have to loose all the new data. Moral? Take your database changes seriously--there is a reason DBA's are sometimes considered assholes. They have to be--they can't just roll back mistakes in the database like you can roll back your buggy code.

Answer 5

While you are certainly asking a question that covers a lot of ground, there are 3 major parts you need to have in place:

1. Use a version control system. Something like Subversion or GIT is going to automatically let you roll back the code of your application to any point in time, or to any point at which you "tagged" your source code. As long as everyone only commits code that builds and runs successfully, every commit will be a valid point to which you can roll changes back to. Your version control system will manage multiple lines of code for you as well with branches. There are many strategies for handling branches, find one that works best for your process.

2. Use a build server. A build server like cruise control, Hudson, or TeamCity is going to automatically ensure that every commit is a "good" commit that compiles and can run successfully (assuming you have tests in place)

3. Include your database schema and static database data with your code in version control. There are tools like LiquiBase that allow you to manage your database structure and are designed to work with multiple developers working concurrently, even across multiple branches. Your code is no good without the corresponding database structure, so you do need to make sure you keep both in sync. Storing your database changes in your source code repository is the easiest way to do that. That being said, you cannot store your full database in your repository. It is too big and it changes too much for your source control's diff/merge support to handle. Depending on your industry, it may also not be legal due to government regulations. If you find you need to roll back your production database due to a bad release, you will need to determine what SQL statements would best undo the changes applied by looking at your stored update scripts.