not exactly fifo

Question

I'm busting my head but cannot find a solution for this. Please consider this scenario:

I have writers that want to write to non-blocking "queue" to another machine on the local network and have a reader that will read data (blocking mode if there are no writers) and do some job A and then return after a long hour and take next data.

So the scenario is like that:

• Writer writes
• Writer writes
• Writer writes
• Writer writes
• Writer writes

• Reader reads and does job

  In the same time while reader is busy:

• Writer writes
• Writer writes
• Writer writes
• Writer writes
• Writer writes
• etc ...

I thought I could do this with a tcp daemon as a reader, but that would mean that it would run simultaneously with fork(s) and I want the reader to process one at a time because it will do a cpu hungry job.

I thought about having a tcp server get the requests and then signal to a FIFO and have another daemon read from the FIFO but it has the same limitations.

I mean the FIFO must read when the writer writes, and I want the writer to write many times faster than the reader.

A db solution would be ok, but a) it is not very fast and b) there is no locking for the reader..I wouldn't want to implement it with sleep(x) it seems not a good programming technique.

Any solutions?

Answer 1

This sounds as if you've got a producer-consumer problem. Take a look at the various implementations in the Wikipedia article and see if one of them meets your needs.

Answer 2

Multithreading seems like the path to follow. Create a thread that starts either the reading or the writing part and use the other thread on the other task. You will need to provide some thread safe communication among threads if you need to pass data from reader to writer. You may also consider using more than one thread for the writing, depending on the context of the application.

With plain C on a POSIX system, pthreads is the way to go.

Answer 3

one option is to have a server (writer) and a client node. this outlines the the general approach:

Server produces jobs and push them into a local queue:

// server thread
while(true)
{
     job = generate();
     jobs_queue.push(job); // push job to a local queue on the server
}

when a client connects to the server, a thread on the server reads everything in the queue and push it to the client. the queue is required to hold jobs while there is no connected client. may be irrelevant in your case.

// server acceptor
while(true)
{
     c = wait_for_connection();
     while(connected(c))
     {
          while(queue.size() >  0)
              c.write(queue.pop()); // send a job to the client

          // block till queue is not empty. can be achieved with sleep or by using a mutex.
     }
}

client node will sit on a tcp socket, reading jobs and putting them into a local queue (on the client). there is a client thread that work like this:

// client thread that poll from server
while(true)
{
    job = readNextJob(); // tcp, blocks if there is nothing to read
    queue.push(job);
}

// client thread that spawn jobs from queue
while(true)
{
    job = queue.pop(); // blocks if queue empty
    job.execute();
    job.waitForCompletion();
}