Common practices if we discover a problem after deploying a web application?

Question

I recently have a problem that my java code works perfectly ok on my local machine, however it just wouldn't work when I deploy it onto the web server, especially the DB part. The worst part is that the server is not my machine. So I had to come back and forth to check the versions of softwares, the db accounts, the settings, and so on...

I have to admit that I did not do a good job with the logging mechanism in the system. However as an newbie programmer with little experience, I had to accept my learning curves. Therefore, here comes a very general but important question:

According to your experience, where would it be most likely to go wrong when it is working perfectly on the development machine but totally surprises you on the production machine?

Thank you for sharing your experience.

Answer 1

A production machine will likely miss some of the libraries and tools you have on your development machine. Or there may be older versions of them. Under circumstances it may interfere with the normal software function.

Database connection situation may be different, meaning users and roles and access levels.

Answer 2

You're most likely running under a different user account. So the environment that you inherit as a developer will be vastly different from that a a production user (which is likely to be a very cut down environment). Your PATH/LD_LIBRARY_PATH (or Windows equivalents) will be different. Permissions will have changed etc. Plus the installed software will be different.

I would strongly recommend maintaining a test box and a test user account that is set up with the same software, permissions and environments as the production user. Otherwise you really can't guarantee anything. You really need to manage and control the production and test servers wrt. accounts/installed software etc. Your development box will always be different, but you need to be aware of the differences.

Finally a deployment sanity check is always a good idea. I usually implement a test URL that can be checked as soon as the app is deployed. It will perform database queries or whatever other key functions are required, and report unambiguously as to what's working/not working via a traffic light mechanism.

Answer 3

In my experience there is no definite answer to this question. Following are some of the issues I faced.

1. Automatic updates was not turned on in dev server (windows) and it was turned on in the production server(which in first place is wrong!). So one of my web application crached due to a patch applied.

2. Some batch jobs were running in the production app server which changed some data on which my application was using.

3. It is not me who does the deployment for my company so most of the time people who deploy miss some registry entries, or add wrong registry entries. Simple but very hard to detect (may be for me ;-) ) once I took hours to identify a space in one of the registry values. Now We have a very long release document which has all the details about all servers used by the application and there is a check list for "current release" which the engineers who deploy the application fill in.

Willl add more if I remeber any.

Answer 4

The absolute number one cause of problems which occur in production but not in development is Environment.

Your production machine is, more likely than not, configured very differently from your development machine. You might be developing your Java application on a Windows PC whilst deploying to a Linux-based server, for example.

It's important to try and develop against the same applications and libraries as you'll be deploying to in production. Here's a quick checklist:

1. Ensure the JVM version you're using in development is the exact same one on the production machine (java -version).
2. Ensure the application server (e.g. Tomcat, Resin) is the same version in production as you're using in development.
3. Ensure the version of the database you're using is the same in production as in development.
4. Ensure the libraries (e.g. the database driver) installed on the production machine are the same versions as you're using in development.
5. Ensure the user has the correct access rights on the production server.

Of course you can't always get everything the same -- a lot of Linux servers now run in a 64-bit environment, whilst this isn't always the case (yet!) with standard developer machines. But, the rule still stands that if you can get your environments to match as closely as possible, you will minimise the chances of this sort of problem.

Ideally you would build a staging server (which can be a virtual machine, as opposed to a real server) which has exactly (or as close as possible to) the same environment as the production server.

If you can afford a staging server, the deployment process should be something like this:

• Ensure application runs locally in development and ensure all unit and functional tests pass in development
• Deploy to staging server. Ensure all tests pass.
• Once happy, deploy to production

Answer 5

One common (albeit easy to detect) problem is conflicting libraries, especially if you're using Maven or Ivy for dependency management and don't double check all the managed dependencies at least once before deploying.

We've had numerous incompatible versions of logging frameworks and even Servlet/JSP API .jar:s a few times too many in our test deployment environment. Also it's always a good idea to check what the shared libraries folder of your tomcat/equivalent contains, we've had some database datasource class conflicts because someone had put postgre's jdbc jar to the shared folder and project came with its own jar for jdbc connectivity.

Answer 6

I always try to get an exact copy of the Server my product is running. After some apps and of course a lot of Bugs i vreated myself a List of common Bugs/Hints. Another Solution i tested for my last project was to get the running Software on that Server and try to configure it. Strange effects can happen with that^^

Last but not least..i always test my apps on different machines.

Answer 7

Beyond just a staging server another strategy for making sure the environments you deploy into are the same is to make sure they are set up automatically. That is you use a tool like Puppet to install all the dependencies that the server has and run your install process before every installation so that all the configuration is reset. That way you can ensure the configuration of the box is what you have set it to during the development process and have the configuration of the production environment in source control.

Answer 8

Specifically you can check all the configuration files (*.xml / *.properties) in your application and ensure that you are not hard coding any paths/variables in your app.

You should maintain different config files for each env. and verify the installation guide from env admin. (if exists)

Other than that versions of all softwares/dependency list etc as described by others.