Implementing a logging library in .NET with a database as the storage medium

Question

I'm just starting to work on a logging library that everyone can use to keep track of any sort of system information while the user is running our application. The simplest example so far is to track Info, Warnings, and Errors.

I want all plugins to be able to use this feature, but since each developer might have a different idea of what's important to report, I want to keep this as generic as possible.

In the C++ world, I would normally use something like a stl::pair<string,string> to act as a key value pair structure, and have a stl::list of these to act as a "row" in the log. The log cache would then be a list<list<pair<string,string>>> (ugh!). This way, the developers can use a const string key like INFO, WARNING, ERROR to have a consistent naming for a column in the database (for SELECTing specific types of information).

I'd like the database to be able to deal with any number of distinct column names. For example, John might have an INFO row with a column called USER, and Bill might have an INFO row with a column called FILENAME. I want the log viewer to be able to display all information, and if one report doesn't have a value for INFO / FILENAME, those fields should just appear blank. So one option is to use List<List<KeyValuePair<String,String>>>, and the another is to have the log library consumer somehow "register" its schema, and then have the database do an ALTER TABLE to handle this situation. Yet another idea is to have a table that's just for key value pairs, with a foreign key that maps the key value pairs back to the original log entry.

I obviously don't want logging to bog down the system, so I only lock the log cache to make a copy of the data (and remove the already-copied data), then a background thread will dump the information to the database.

My specific questions regarding this are:

1. Do you see any performance issues? In other words, have you ever tried something like this and found that certain things just don't work well in practice?
2. Is there a more .NETish way to implement the key value pairs, other than List<List<KeyValuePair<String,String>>>?
3. Even if there is a way to do #2 better, is the ALTER TABLE idea I proposed above a Bad Thing?
4. Would you recommend multiple databases over a single one? I don't yet have an idea of how frequently the log would get written to, but we ideally would like to have lots of low level information. Perhaps there should be a DB with a fixed schema only for the low level stuff, and then another DB that's more flexible for reporting information back to users.

Answer 1

why don't you check log4net? It may be enough for your purposes and you would avoid re invent a wheel already invented many times :-)

Here you have some configuration examples about how to store the logging information on the database:

http://logging.apache.org/log4net/release/config-examples.html

Answer 2

1. Not until there are some (Knuth).
2. Create models that actually model your log entries, then cache them in a List
3. I'd provide standard fields and store all user configurables as XML. Allows for a flexible logging system, doesn't result in schema alterations, and is (at least in Sql Server) searchable.
4. More databases is more maintenance. Keep it simple. In fact, just use Log4net.

Answer 3

Like others have already noted, there are several popular logging frameworks that have a lot of built-in functionality. While none of them have the flexibility you desire, my experience is that you never really need that kind of flexability. It's only logging :-).

Here is a list of some common logging libraries:

• log4net
• NLog
• Enterprise Library
• Termite
• ELMAH
• CuttingEdge.Logging
• (did I miss one?)

And when you have trouble chosing one, use a logging facade to hide the implementation. For this you can pick:

• Common.Logging
• Simple Logging Facade.