Limit Derby Log File Size

Question

Hi All,

Our app sends the contents of the derby.log file to our server whenever Apache Derby throws a SQLException in our app.

In order to get detailed logs, we are setting the 'derby.infolog.append' property to true.

However, we are noticing enormously large logs files since the logs also contain bootup output each time a connection is made to the database.

NOTE: we are using Derby in embedded mode.

Is there a way to have derby limit the total number of lines it logs to derby.log file?

For example, only logging the most recent 1000 lines of logs and then begin to overwrite the oldest entries.

Our goal is to get useful debugging info from end users but to prevent the log files from growing to unmanageable sizes.

Thanks in advance,

Jim

Answer 1

I'm not that familiar with derby but I couldn't find an "easy" way to do this.

But there are some derby properties you could set to implement this yourself.

Check these

derby.stream.error.field

derby.stream.error.file

derby.stream.error.method

derby.stream.error.logSeverityLevel

So I imagine you writing some class which subclasses java.io.OutputStream or java.io.Writer and then you either

• implements the wanted behaviour or
• do it similar to How do I limit the size of log file? + wrap as one of the above or
• you ripp-off get some ideas for a RollingFileLoggerClass from some other project (RollingFileAppender log4j, RollingFileWriter clapper, ...)

Answer 2

Another way to handle this would be to write your own code which rotates, truncates, compresses, or otherwise pares down the derby.log file in-between runs of Derby.

You don't mention what version of Derby you're running, but I thought the line-per-connection output was removed in a more recent release. Or perhaps it was only removed from Network Server output rather than from derby.log output?

If it's the line-per-connection output that is swelling your derby.log, then you might consider using connection pooling techniques so that you don't make so many connections. Generally you can hang on to connections for the lifetime of your application; you don't have to create and destroy them very often.

If you think there is excess unnecessary output going to derby.log, you might go log an enhancement request at the Derby community bug tracker with examples, to ensure that future versions of Derby don't log unneeded stuff.

Answer 3

You can create a custom logging class, and specify this using derby.stream.error.field as mentioned above. The logging class doesn't have to implemented as a file - if you will be limiting the size of the logging data you can easily hold it in memory.

The second advantage to this is that when a problem is encountered, you have a great deal of flexibility in what to do with the logging data. Perhaps compress (or encrypt) the data and automatically open a ticket in your help system (as an example).

Here's an example of a very simple custom logging solution:

import java.io.CharArrayWriter;

public class CustomLog {

    public static CharArrayWriter log = new CharArrayWriter();

    public static void dump() {
        System.out.println(log.toString());
    }
}

You can replace the CharArrayWriter with a size limited buffer of some sort, and add an implementation of dump() to do what you will with the resulting log data.

A short example program demonstrating this follows:

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.sql.Statement;

public class DerbyLoggingExample {

    public DerbyLoggingExample() {
        System.setProperty( "derby.stream.error.field", "CustomLog.log");

        String driver = "org.apache.derby.jdbc.EmbeddedDriver";
        String dbName = "logdemoDB";
        String connectionURL = "jdbc:derby:" + dbName + ";create=true";

        String createCommand = "create table test_table ("
            + "test_id int not null generated always as identity, "
            + "test_name varchar(20)"
            + ")";

        try {
            Class.forName(driver);
        }
        catch( java.lang.ClassNotFoundException e ) {
            System.out.println( "Could not load Derby driver." );
            return;
        }

        Connection conn = null;
        Statement statement = null;

        try {
            conn = DriverManager.getConnection(connectionURL);
            statement = conn.createStatement();

            statement.execute(createCommand);
        }
        catch( SQLException sqle ) {
            sqle.printStackTrace();
            System.out.println( "SQLException encountered. Dumping log.");
            CustomLog.dump();
            return;
        }
        finally {
            try {
                statement.close();
                conn.close();
            }
            catch( SQLException e ) {
                // Do nothing.
            }
        }

        System.out.println( "Processing done. Dumping log." );
        CustomLog.dump();
    }

    public static void main(String[] argv) {
        DerbyLoggingExample thisApp = new DerbyLoggingExample();
    }
}