C# high speed data logging data handling

Question

I've written an application that logs trace data from an embedded system via UDP. Currently I receive datagrams and parse out the variable length records and store them in a list. The front end can access the list and present the data (graphs and text lists etc).

The problem I'm running into is that sometimes I need to log an exceptional amount of data. So much that my list implementation causes an out of memory exception.

My requirements are:

• Allow multithreaded reading and writing of the data (can't just post process)
• Handle large amounts of data (worst case ~2MB/s ... 7.2GB/hr of logging)
• Allow storage of data set
• Random read, index based, access

Does anyone have some suggestions about how to attack this? Here were a few thoughts that I had:

• I'd like a nifty disk backed, memory cached List. It seems like something that would exist but I haven't found one.
• Local database? I don't know too much about databases but it seems like overkill.
• Store the data to a file right away. Keep a list in memory that holds the byte offset for each record index. Can my reader thread access this simultaneously?

Thanks! Kurt

Answer 1

A local database really would be a nice way to handle this - especially because queries would aid with your investigation of logs. Plus, then your UDP receiving program could just be a separate thread that spits information at the database (if your data's REALLY fast-paced, you could have two buffers and alternate between them; flush the full buffer to the database while the other one is filling up). It really depends on the scale of your project though.

You could always use your third option (storing to a file right away), and have a separate "Log Investigation" tool that reads that file in without running into OOM exceptions.

Answer 2

.NET 4 has Lock free queues. You could setup a queue in which one thread is adding things from the UDP component to log and another thread is consuming these logs and putting them into a file or a database.

Answer 3

What I have is a queue that I add my message I'm logging to with the Log(string contents) method. I have another method that I start in a background thread which is constantly reading the queue and writing to the file. Timestamps are preserved even though the writing can be done after the fact if too much data comes through.

The logging method is static and public so it can be called from any thread. I can't guarantee this code compiles because I ripped it out of my project and removed some stuff.

I've found that I don't receive a performance boost with more than 1 or 2 threads writing because my computers DISK I/O Sucks. You might be able to speed this up if you split out your logging to several files, can't say for sure though.

private static StreamWriter sw;
private static Queue<string> logQueue = new Queue<string>();
public static string logLock = "";
public static void LogLoop()
{
    sw = new StreamWriter("logFilePath.log"), true);
    sw.AutoFlush = true;
    while (true)
    {
     while (logQueue.Count > 0)
     {
      string s = "";
      lock (logLock) // get a lock on the queue
      {
       s = logQueue.Dequeue();
      }
      sw.WriteLine(s);       
     }
     Thread.Sleep(10);
    }
}
public static void Log(string contents)
{
    contents = DateTime.Now.ToString("MM-dd-yy - HH:mm:ss ffff") + " - " + contents; // add a timestamp

    lock (logLock) // get a lock on the queue
    {
     logQueue.Enqueue(contents);
    }
}

This is how I start my background thread method.

Thread logThread = new Thread(LogLoop);
logThread.IsBackground = true;
logThread.Name = "Logging Thread";
logThread.Start();

Answer 4

I am using Josiah's approach to create a reusable Logger class. However, I use a flag instead of while(true) that allows the loop to terminate when set to false.

while (logging)  // instead of while(true)
{
    while (logQueue.Count > 0)
    {
        string s = "";
        lock (logLock)
        {
           s = logQueue.Dequeue();
        }
        write(s);
    }
    Thread.Sleep(timer);
}

It works well, but I've found that it's possible to enqueue thousands of messages before the logQueue.Count value actually changes.

for (int i = 0; i <5000; i++)
{
     lock (logLock)
     {
       logQueue.Enqueue(i.toString());
     }
}
logging = false;

Sometimes the above code causes the LogLoop to terminate before any messages are actually written to the file. Putting a pause before setting logging to false fixes it, but I'm still surprised that logQueue.Count does not always change before the queue recognizes the messages.