Architectural Suggestions in a Linux App

Question

Hi, I've done quite a bit of programming on Windows but now I have to write my first Linux app.

I need to talk to a hardware device using UDP. I have to send 60 packets a second with a size of 40 bytes. If I send less than 60 packets within 1 second, bad things will happen. The data for the packets may take a while to generate. But if the data isn't ready to send out on the wire, it's ok to send the same data that was sent out last time. The computer is a command-line only setup and will only run this program.

I don't know much about Linux so I was hoping to get a general idea how you might set up an app to meet these requirements. I was hoping for an answer like:

Make 2 threads, one for sending packets and the other for the calculations.

But I'm not sure it's that simple (maybe it is). Maybe it would be more reliable to make some sort of daemon that just sent out packets from shared memory or something and then have another app do the calculations? If it is some multiple process solution, what communication mechanism would you recommend? Is there some way I can give my app more priority than normal or something similar?

PS: The more bulletproof the better!

Answer 1

Two threads as you've suggested would work. If you have a pipe() between them, then your calculating thread can provide packets as they are generated, while your comms thread uses select() to see if there is any new data. If not, then it just sends the last one from it's cache.

I may have over simplified the issue a little...

Answer 2

I've done a similar project: a simple software on an embedded Linux computer, sending out CAN messages at a regular speed.

I would go for the two threads approach. Give the sending thread a slightly higher priority, and make it send out the same data block once again if the other thread is slow in computing those blocks.

60 UDP packets per second is pretty relaxed on most systems (including embedded ones), so I would not spend much sweat on optimizing the sharing of the data between the threads and the sending of the packets.

In fact, I would say: keep it simple! I you really are the only app in the system, and you have reasonable control over that system, you have nothing to gain from a complex IPC scheme and other tricks. Keeping it simple will help you produce better code with less defects and in less time, which actually means more time for testing.

Answer 3

The suggestion to use a pair of threads sounds like it will do the trick, as long as the burden of performing the calculations is not too great.

Instead of using the pipe() as suggested by Cogsy, I would be inclined to use a mutex to lock a chunk of memory that you use to contain the output of your calculation thread - using it as a transfer area between the threads.

When your calculation thread is ready to output to the buffer it would grab the mutex, write to the transfer buffer and release the mutex.

When your transmit thread was ready to send a packet it would "try" to lock the mutex. If it gets the lock, take a copy of the transfer buffer and send it. If it doesn't get the lock, send the last copy.

You can control the priority of your process by using "nice" and specifying a negative adjustment figure to give it higher priority. Note that you will need to do this as superuser (either as root, or using 'sudo') to be able to specify negative values.

---

edit: Forgot to add - this is a good tutorial on pthreads on linux. Also describes the use of mutexes.

Answer 4

I didn't quite understand how hard is your 60 packets / sec requirement. Does a burst of 60 packets per second fill the requirement? Or is a sharp 1/60 second interval between each packet required?

This might go a bit out of topic, but another important issue is how you configure the Linux box. I would myself use a real-time Linux kernel and disable all unneeded services. Other wise there is a real risk that your application misses a packet at some time, regardless of what architecture you choose.

Any way, two threads should work well.

Answer 5

I posted this answer to illustrate a quite different approach to the "obvious" one, in the hope that someone discovers it to be exactly what they need. I didn't expect it to be selected as the best answer! Treat this solution with caution, because there are potential dangers and concurrency issues...

You can use the setitimer() system call to have a SIGALRM (alarm signal) sent to your program after a specified number of milliseconds. Signals are asynchronous events (a bit like messages) that interrupt the executing program to let a signal handler run.

A set of default signal handlers are installed by the OS when your program begins, but you can install a custom signal handler using sigaction().

So all you need is a single thread; use global variables so that the signal handler can access the necessary information and send off a new packet or repeat the last packet as appropriate.

Here's an example for your benefit:

#include <stdio.h>
#include <signal.h>
#include <sys/time.h>
int ticker = 0;

void timerTick(int dummy)
{
    printf("The value of ticker is: %d\n", ticker);
}

int main()
{
    int i;

    struct sigaction action;
    struct itimerval time;

    //Here is where we specify the SIGALRM handler
    action.sa_handler = &timerTick;
    sigemptyset(&action.sa_mask);
    action.sa_flags = 0;

    //Register the handler for SIGALRM
    sigaction(SIGALRM, &action, NULL);

    time.it_interval.tv_sec = 1;       //Timing interval in seconds
    time.it_interval.tv_usec = 000000; //and microseconds
    time.it_value.tv_sec = 0;  //Initial timer value in seconds
    time.it_value.tv_usec = 1; //and microseconds

    //Set off the timer
    setitimer(ITIMER_REAL, &time, NULL);

    //Be busy
    while(1)
     for(ticker = 0; ticker < 1000; ticker++)
      for(i = 0; i < 60000000; i++)
       ;
}

Answer 6

Two threads would work, you will need to make sure you lock your shared data structure through so the sending thread doesn't see it half way through an update.

60 per second doesn't sound too tricky.

If you are really concerned about scheduling, set the sending thread's scheduling policy to SCHED_FIFO and mlockall() its memory. That way, nothing will be able to stop it sending a packet (they could still go out late though if other things are being sent on the wire at the same time)

There has to be some tolerance of the device - 60 packets per second is fine, but what is the device's tolerance? 20 per second? If the device will fail if it doesn't receive one, I'd send them at three times the rate it requires.

Answer 7

I would stay away from threads and use processes and (maybe) signals and files. Since you say "bad things" may happen if you don't send, you need to avoid lock ups and race conditions. And that is easier to do with separate processes and data saved to files.

Something along the line of one process saving data to a file, then renaming it and starting anew. And the other process picking up the current file and sending its contents once per second.

Unlike Windows, you can copy (move) over the file while it's open.

Answer 8

Follow long-time Unix best practices: keep it simple and modular, decouple the actions, and let the OS do as much work for you as possible.

Many of the answers here are on the right track, but I think they can be even simpler:

• Use two separate processes, one to create the data and write it to stdout, and one to read data from stdin and send it. Let the basic I/O libraries handle the data stream buffering between processes, and let the OS deal with the thread management.

• Build the basic sender first using a timer loop and a buffer of bogus data and get it sending to the device at the right frequency.

• Next make the sender read data from stdin - you can redirect data from a file, e.g. "sender < textdata"

• Build the data producer next and pipe its output to the sender, e.g. "producer | sender".

Now you have the ability to create new producers as necessary without messing with the sender side. This answer assumes one-way communication.

Keeping the answer as simple as possible will get you more success, especially if you aren't very fluent in Linux/Unix based systems yet. This is a great opportunity to learn a new system, but don't over-do it. It is easy to jump to complex answers when the tools are available, but why use a bulldozer when a simple trowel is plenty. Mutex, semaphores, shared memory, etc, are all useful and available, but add complexity that you may not really need.

Answer 9

I agree with the the two thread approach. I would also have two static buffers and a shared enum. The sending thread should have this logic.

loop
    wait for timer
    grab mutex
    check enum {0, 1}
    send buffer 0 or 1 based on enum
    release mutex
end loop

The other thread would have this logic:

loop
    check enum
    choose buffer 1 or 0 based on enum (opposite of other thread)
    generate data
    grab mutex
    flip enum
    release mutex
end loop

This way the sender always has a valid buffer for the entire time it is sending data. Only the generator thread can change the buffer pointer and it can only do that if a send is not in progress. Additionally, the enum flip should never take so many cycles as to delay the higher priority sender thread for very long.

Answer 10

Thanks everyone, I will be using everyones advice. I wish I could select more answers than 1!

For those that are curious. I dont have source for the device, its a propietary locked down system. I havent done enough testing to see how picky the 60 packets a second is yet. Thats all their limited docs say is "60 packets a second". Due to the nature of the device though, bursts of packets will be a bad thing. I think I will be able to get away with sending more than 60 a second to make up for the occasional missed packets..