How to setup a central logging server and divide logging events from test and production?

Question

How best should I setup a central logging (or perhaps 2, each one exclusive to either test or prod) server such that:

I do not have to worry about the code referencing any conditionals that might accidentally log non-prod errors into the production error log4j bus. In other words, I do not wants to have code such as:

if (!production)
  logWrapper.logTest(error);
else 
  logWrapper.logProd(error);

I do not want this because it tightly couples something such as the server name, references it, and now the code can only work on one of two server names: prod and test. Nor can I declaratively set this in a config because of the probability that a developer will not set this correctly and pollute the logging repositories. Any Suggestions?

A few details, some aren't going to factor in but I list them anyway:

• Only web applications, running on Oracle Application Server 10.1.3.4 now, and in the future Oracle WebLogic
• going to be using a central logging server as demonstrated in this example: http://timarcher.com/?q=node/10 ...EXCEPT that the central server will log text files and not perform emailing!
• Will use log4j and maybe it will be wrapped via slf4j (the slf4j is kind of irreverent to mention so ignore it for the most part..)

Answer 1

Use different logging configuration files for production and staging.

You could do this a couple different ways

1) Keep in your app two different configuration files (one for stage and one for production) and have your application have some startup code that knows which configuration file to use based on the machine it is on or maybe some setting in the applications configuration file that sets an environment setting.

2) Simply have one file but swap out with the appropriate file when testing or in production.

Answer 2

I really think that you need to set this in a config. It's a correct decision.

Answer 3

Without a lot of details about you're environment it's hard to give a specific answer, but if I was setting up logging to handle multiple environments here's the approach I'd take.

1) Applications should not care about logging. For example the following should work in all environments:

logger.warn("message", error);

2) Create a log4j (or whatever logging framework) configuration to configure logging. This can be done one of two ways. The first way is to create a test_log4j.{xml,properties}, prod_log4j.{xml,properties}, etc per environment. This keeps configuration for each environment very straightforward and to add a new environment is just adding a new file. The alternative is to create one log4j.{xml,properties} and use the ant style property replacement to insert the appropriate values per environment. The property values can be picked up from environment specific property files or from -Dkey=value arguments passed to the JVM. This is more of the DRY style of configuration assuming the configuration for each environment is pretty similar. As an example to change a logging level between environments:

<logger name="my.example">
  <level value="${my.example.level}"/>
</logger>

3) Create a start script that determines the environment (probably based on the machine or some file on the machine) and then loads the correct configuration. This can either be the correct log4j configuration, correct property files or correct -Dkey=value arguments.

Answer 4

You didn't mention what RDBMS you're using. If you're using Oracle, you can use the SYS_CONTEXT() function to get the database name. From there, you can determine if you're in production/dev/qa/test/etc.

Here's a nice article on sys_context()

Answer 5

This particular issue is addressed in logFaces, in fact it's one of the major problems it was design to solve - centralize and reduce log flooding. In your application config. you specify application name, for example, "My production app". All log statements from the application will then flow into centralized log server and will be available in storage for queries and in real-time viewing. You don't change your code, it's only a matter of configuration. Usually we have several instances of same application in development and several versions in QA, each developer can tune it's viewer to fetch what is going on on others. I often have a watch on QA hosts and know there is a problem even before they realize it. Another interesting scenario is when you want to see what is going on in all applications at once but focusing only on, say, data layer.

Answer 6

mattkemp pretty much nailed this one. The one thing I might add is that the key difference I'd expect between production and test is that the test-version probably would have debug-logging permanently active, while in prod you would only activate debug-logging on rare occasions of strange behavior.

Other than that, at least I would expect roughly the same types of log entries from test and production, since whatever can happen during testing can also happen during production (and will).

When it comes to the server setup itself, I'd say just go for either syslog or the windows NT event log, with the appropriate log4j setup. That way you can also log other parts of the final systems in the same logging server. A good logging system would then show on which server the event was generated, so it should be fairly clear whether the event came from prod or testing.

For a good web-based syslog gui, you can check out http://www.phplogcon.org/