How to regulate the amount of printouts generated by a logging instruction produces over time?

Question

How can I limit a program log printouts to a maximum of X printouts within Y seconds?

Programming server side with java.util.logging, my code has a lot of info, warning, and error statements like:

s_logger.logp(Level.WARNING, myClassName, myMethodName, "msg.code.in.properties.file");

On the one hand, I do want to see the warning message above printed to the STDOUT, since it serves an indication that something went wrong, especially when investigating problems in production, but on the other hand, printing numerous lines that tells the same story over a short period of time, has no added value but instead hit performance and introduces scrolling blindness.

What I'm looking for is a mechanism/API that could regulate the printouts to no more than x messages, per message code, during y seconds. For example, over the course of a minute, I don't want to generate and write to the log more than 10 messages of the type: "transaction timed out".

I'd be happy for a API or a library reference, Anyone?

Answer 1

if you want to use standard logging library/mechanism, you may produce your own wrapper around the standard logger which will check for your conditions whether to log or not, and pass the info down to the logging mechanism only if conditions are satisfied.

or you can write your own (simple, or maybe not) logger :)

Answer 2

Thanks. I already considered that.
I was hoping for something that's already out there, since I believe this problem to be common on server side development projects.

Answer 3

Gili, I'm not sure anything like this exists right out of the box. A good logging implementation like Logback will let you define Loggers for each portion of your code and then configure the logging level of each Logger (eg, only WARN and above messages by default, but INFO or even FINE for the Logger controlling an area you're actively debugging).

A logging implementation also typically can manage multiple log files, rolling over the current log into a compressed archive version and starting a fresh one either at regular intervals or whenever a log file grows too large.

If you are in control of the code, you could modify your logging statements to not generate so many redundant warnings. For example if you are getting a lot of transaction timeouts, maybe you could wait for successively longer periods between transaction start attempts before giving up altogether. (This has the advantage of ratcheting down the load on the servers implementing the transactions, giving them time to recover.)

Another approach alluded to by zappan is to create your own log filters. In java.util.logging you can attach your own Filters to your Filters to restrict what they print out.

A more elaborate filtering approach would be to configure your logging implementation to send all log messages out on a multicast address instead of a file. You'd write a filtering program that listens to the multicast and selectively writes the ones you want out to disk (eg, this could run on a log aggregation server if you have a distributed system). Other filters could pull out the most important messages and update a graphical display of your system's health.