Splitting log4j Output with Quartz Worker Threads

Question

I'm working on an application that consists of an overall Quartz-based scheduler and "CycledJob" run using CronTriggers. The purpose of the application is to process inputs from different email inboxes based on the source country.

Based on the country that it comes in from (i.e. US, UK, FR, etc.) the application triggers one job thread to run each country's processing cycle, so there would be a UK Worker thread, one for US, France, etc. When formatting the output to log4j, I'm using the thread parameter, so it emits [ApplicationName_Worker-1], [ApplicationName_Worker-2] etc. Try as I might, I can't find a way to name the threads since they're pulled out of Quartz's Thread Pools. Although I could possibly go so far as to extend Quartz, I'd like to work out a different solution instead of messing with the standard library.

Here's the problem: When using log4j, I'd like to have all log items from the US thread output to a US only file, likewise for each of the country threads. I don't care if they stay in one unified ConsoleAppender, the FileAppender split is what I'm after here. I already know how to specify multiple file appenders and such, my issue is I can't differentiate based on country. There are 20+ classes within the application that can be on the execution chain, very few of which I want to burden with the knowledge of passing an extra "context" parameter through EVERY method... I've considered a Strategy pattern extending a log4j wrapper class, but unless I can let every class in the chain know which thread it's on to parameterize the logger call, that seems impossible. Without being able to name the thread also creates a challenge (or else this would be easy!).

So here's the question: What would be a suggested approach to allow many subordinate classes in an application that are each used for every different thread to process the input know that they are within the context of a particular country thread when they are logging?

Good luck understanding, and please ask clarifying questions! I hope someone is able to help me figure out a decent way to tackle this. All suggestions welcome.

Answer 1

I wish I could be a bit more helpful than this, but you may want to investigate using some filters? Perhaps your logging could output the country code and you could match your filter based on that?

A StringMatchFilter should probably be able to match it for you.

Couldn't get the below address to work properly as a link, but if you look at it, it has some stuff on separate file logging using filters.

http://mail-archives.apache.org/mod_mbox/logging-log4j-user/200512.mbox/<1CC26C83B6E5AA49A9540FAC8D35158B01E2968E@pune.kaleconsultants.com > (just remove the space before the >)

http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/spi/Filter.html

Answer 2

I may be completely off base on my understanding of what you are attempting to accomplish, but I will take a stab at the solution. It sounds like you want a separate log file for each country for which you are processing email. Based on that understanding, here is a possible solution:

1. Set up an appender in your log4j configuration for each country for which you wish to log separately (US example provided):

   log4j.appender.usfile=org.apache.log4j.FileAppender

   log4j.appender.usfile.File=us.log

   log4j.appender.usfile.layout=org.apache.log4j.PatternLayout

   log4j.appender.usfile.layout.ConversionPattern=%m%n

2. Create a logger for each country and direct each of them to the appropriate appender (US example provided):

   log4j.logger.my-us-logger=debug,usfile

3. In your code, create your Logger based on the country for which the email is being processed:

   Logger logger = Logger.getLogger("my-us-logger");

4. Determine how you will accomplish step 3 for the subsequent method calls. You could repeat step 3 in each class/method; or you could modify the method signatures to accept a Logger as input; or you could possibly use ThreadLocal to pass the Logger between methods.

Extra info: If you do not want the log statements going to parent loggers (e.g. the rootLogger), you can set their additivity flags to false (US example provided):

log4j.additivity.my-us-logger=false

Answer 3

At the top of each country's processing thread, put the country code into Log4j's mapped diagnostic context (MDC). This uses a ThreadLocal variable so that you don't have to pass the country up and down the call stack explicitly. Then create a custom filter that looks at the MDC, and filters out any events that don't contain the current appender's country code.

In your Job:

...
public static final String MDC_COUNTRY = "com.y.foo.Country";
public void execute(JobExecutionContext context)
  /* Just guessing that you have the country in your JobContext. */
  MDC.put(MDC_COUNTRY, context.get(MDC_COUNTRY));
  try {
    /* Perform your job here. */
    ...
  } finally {
    MDC.remove(MDC_COUNTRY);
  }
}
...

Write a custom Filter:

package com.y.log4j;

import org.apache.log4j.spi.LoggingEvent;

/**
 * This is a general purpose filter. If its "value" property is null, 
 * it requires only that the specified key be set in the MDC. If its 
 * value is not null, it further requires that the value in the MDC 
 * is equal.
 */
public final class ContextFilter extends org.apache.log4j.spi.Filter {

  public int decide(LoggingEvent event) {
    Object ctx = event.getMDC(key);
    if (value == null)
      return (ctx != null) ? NEUTRAL : DENY;
    else
      return value.equals(ctx) ? NEUTRAL : DENY;
  }

  private String key;
  private String value;

  public void setContextKey(String key) { this.key = key; }
  public String getContextKey() { return key; }
  public void setValue(String value) { this.value = value; }
  public String getValue() { return value; }

}

In your log4j.xml:

<appender name="fr" class="org.apache.log4j.FileAppender">
  <param name="file" value="france.log"/>
  ...
  <filter class="com.y.log4j.ContextFilter">
    <param name="key" value="com.y.foo.Country" />
    <param name="value" value="fr" />
  </filter>
</appender>

Answer 4

Why not just call Thread.setName() when your job starts to set the name the Thread? If there is an access problem, configure quartz to use your own thread pool.