Fast data recording/logging on a separate thread in C#

Question

Hello,

We're developing an application which reads data from a number of external hardware devices continuously. The data rate is between 0.5MB - 10MB / sec, depending on the external hardware configuration.

The reading of the external devices is currently being done on a BackgroundWorker. Trying to write the acquired data to disk with this same BackgroundWorker does not appear to be a good solution, so what we want to do is, to queue this data to be written to a file, and have another thread dequeue the data and write to a file. Note that there will be a single producer and single consumer for the data.

We're thinking of using a synchronized queue for this purpose. But we thought this wheel must have been invented so many times already, so we should ask the SO community for some input.

Any suggestions or comments on things that we should watch out for would be appreciated.

Thank you.

Answer 1

Have you tried MSMQ

Cheers

Answer 2

I am pretty confident that your queue will be pretty much ok. But make sure that you use efficient method of storing/retrieving data not to overhaul you logging procedure with memory allocation/deallocation. I would go for some pre-allocated memory buffer, and use it as a circular queue.

Answer 3

u might need queing

eg. code

protected Queue<Byte[]> myQ;
or
protected Queue<Stream> myQ;

//when u got the content try
myQ.Enque(...);

and use another thread to pop the queue

// another thread
protected void Loging(){
 while(true){
  while(myQ.Count > 0){
   var content = myQ.Dequeue();
   // save content
  }
  System.Threading.Thread.Sleep(1000);
 }
}

Answer 4

I would do what a combination of what mr 888 does.

Basicly in you have 2 background workers, one that reads from the hardware device. one that writes the data to disk.

Hardware background worker:
Adds chucks on data from the hardware in the Queue<> . In whatever format you have it in.

Write background worker
Parses the data if needed and dumps to disk.

One thing to consider here is is getting the data from the hardware to disk as fast as posible importent?
If Yes, then i would have the write brackground test basicly in a loop with a 100ms or 10ms sleep in the while loop with checking if the que has data.
If No, Then i would have it either sleep a simular amount ( Making the assumtion that the speed you get from your hardware changes periodicly) and make only write to disk when it has around 50-60mb of data. I would consider doing it this way because modern hard drives can write about 60mb pr second ( This is a desktop hard drive, your enterprice once could be much quicker) and constantly writing data to it in small chucks is a waste of IO bandwith.