How to safely update a live website

Question

We have a fairly simple Django-based website for doing CRUD operations. I've been doing testing and development locally and then checking out releases and database schema changes onto the live server once testing is done. We've recently run into a problem when releasing some types of changes. Imagine the following sequence of events:

1. User opens a web form
2. Site is updated to require new field on this form
3. User submits the form they have been working on
4. Server returns an error because it expected to receive the new field that was added in step 2

How do other sites handle these kinds of problems? My ideas:

• Take the site offline while updates are being made. This doesn't really solve the problem, because a user could have a web form open for an infinite amount of time before submitting it, but after a certain amount of time it would be unlikely that anyone would be submitting the form.
• Do automatic updates at very low traffic times. Again this doesn't really solve the problem, but our site isn't that popular and if we did an update at 3:00a I doubt there would be many users. One concern with this technique is automatic updates that fail.
• Versioning forms so that the server recognizes that an old form is being submitted and provides a more user friendly response. Are there automated tools that could help with this?

Thoughts?

Answer 1

I can't see how you could achieve this without coding in your validation the possibility for an update to have occured. eg. "anticipating" that fields might have changed when submitting and either setting defaults/rejecting accordingly.

What about "scaling back" submission of content as a maintenance window approaches, to minimise impacted users to those who have left their browsers open?

Answer 2

If it's really a big problem, you could include the code version as some sort of hidden variable in each of your forms. If the version submitted doesn't match the currently running version of the application, you could display a proper error, and get them to fill in any new fields which might exist on the form. You could even go a bit further and only display messages for which the form has changed. Possibly create some sort of hash based on the definition of the form, and use that as the hidden field. If the hash is wrong, you know they are submitting an incorrect form.

Answer 3

Changes to a published API (or UI, in this case) is always tricky. If possible, preserve backwards compatibility. For most forms, I would reckon that the functionality wouldn't change between versions. You might add or remove a field or two, but that would be handled by the form validation on the backend. That's essentially what you're describing in your step 4. I don't really consider that much of a problem; Runtime errors happen from time to other - As long as your application handles it gracefully and informs the user of the problem, then no issue really.

Answer 4

There are plenty of sites I use (granted, mostly internal to my place of employment) that announce something like "We'll be down for maintenance this coming weekend from 6:00 p.m. Saturday to 6:00 a.m. Sunday morning. Please plan to be off the system at that time." While it's not perfect, nothing is, and this seems like a good approach. Just carve off plenty of time to put out the new stuff, test it, and roll back to the old if needed. If you feel it's necessary, you could always set up a simple page that says "Sorry, we're not available now" and direct people to that during the down time. Generally nobody will complain if you don't need all the time you originally stated and you're back up early (maybe the slackers who want an excuse to avoid work would be the exception, but they're probably not working then anyway).

Answer 5

The 'correct' way to do it is to have well defined views that handle this entire class of failure gracefully. In the case of adding a new field to your model that is required (I'm presuming that's what's happening), the view should deal with that with a ValidationError exception that throws a friendly error message and sends the user back to the form (which, when reloaded, should have the new field available). No matter what fields are added to the model, the exception will throw a clean error and send the user back.