views:

249

answers:

4

I have a system that needs at least 10 mseconds of accuracy for timers.
I went for timerfd as it suits me perfectly, but found that even for times up to 15 milliseconds it is not accurate at all, either that or I don't understand how it works.

The times I have measured were up to 21 mseconds on a 10 mseconds timer.
I have put together a quick test that shows my problem.
Here a test:

#include <sys/timerfd.h>
#include <time.h>
#include <string.h>
#include <stdint.h>

int main(int argc, char *argv[]){

    int timerfd = timerfd_create(CLOCK_MONOTONIC,0);
    int milliseconds = atoi(argv[1]);
    struct itimerspec timspec;
    bzero(&timspec, sizeof(timspec));
    timspec.it_interval.tv_sec = 0;
    timspec.it_interval.tv_nsec = milliseconds * 1000000;
    timspec.it_value.tv_sec = 0;
    timspec.it_value.tv_nsec = 1;

    int res = timerfd_settime(timerfd, 0, &timspec, 0);
    if(res < 0){
       perror("timerfd_settime:");
    }
    uint64_t expirations = 0;
    int iterations = 0;
    while( res = read(timerfd, &expirations, sizeof(expirations))){
        if(res < 0){ perror("read:"); continue; }
        if(expirations > 1){
            printf("%lld expirations, %d iterations\n", expirations, iterations);
            break;
        }
        iterations++;
    }
}

And executed like this:

Zack ~$ for i in 2 4 8 10 15; do echo "intervals of $i milliseconds"; ./test $i;done
intervals of 2 milliseconds
2 expirations, 1 iterations
intervals of 4 milliseconds
2 expirations, 6381 iterations
intervals of 8 milliseconds
2 expirations, 21764 iterations
intervals of 10 milliseconds
2 expirations, 1089 iterations
intervals of 15 milliseconds
2 expirations, 3085 iterations

Even assuming some possible delays, 15 milliseconds delays sounds too much for me.

A: 

Depending on what else the system is doing, it may be a bit slow in switching back to your task. Unless you have a "real" realtime system, there's no guarantee it will do better than what you're seeing, although I agree that result is a bit disappointing.

You can (mostly) eliminate that task switch / scheduler time. If you have CPU power (and electrical power!) to spare, a brutal but effective solution would be a busy wait spin loop.

The idea is to run your program in a tight loop that continuously polls the clock for what time it is, and then calls your other code when the time is right. At the expense of making your system act very sluggish for everything else and heating up your CPU, you will end up with task scheduling that is mostly jitter free.

I wrote a system like this once under Windows XP to spin a stepper motor, supplying evenly spaced pulses up to 40K times per second, and it worked fine. Of course, your mileage may vary.

Carl Smotricz
Why didn't you just get a cheap micro like a PIC or Amtel to do the legwork? Nowadays I'd use an Ardiuno to do the stepper control, and use the USB connection to send faster/slower/start/stop signals to it.
Emyr
+1  A: 

Here's a theory. If HZ is set to 250 for your system ( as is typical ) then you have a 4 millisecond timer resolution. Once your process is swapped out by the scheduler, it's likely that a number of other processes will be scheduled and run before your process gets another time slice. This might explain you seeing timer resolutions in the 15 to 21 millisecond range. The only way to get around this would be to run a real-time kernel.

The typical solution for high resolution timing on non-realtime systems is basically to busy wait with a call to select.

Robert S. Barnes
I don't think many desktop oriented distros set HZ to 250, and most of them these days turn on tickless operation so HZ shouldn't matter much for timing. Most setups these days you should generally be able to get millisecond granularity without a realtime kernel easily, and you don't need the RT Linux patches unless your realtime load is really really tight...
Spudd86
Mainline Linux IS a realtime system these days, it's that not many people know it :P (well it CAN be if you write your code to take advantage of the realitme features)
Spudd86
+3  A: 

Try altering it as follows, this should pretty much garuntee that it'll never miss a wakeup, but be careful with it since running realtime priority can lock your machine hard if it doesn't sleep, also you may need to set things up so that your user has the ability to run stuff at realtime priority (see /etc/security/limits.conf)

#include <sys/timerfd.h>
#include <time.h>
#include <string.h>
#include <stdint.h>
#include <stdio.h>
#include <sched.h>

int main(int argc, char *argv[]) 
{
    int timerfd = timerfd_create(CLOCK_MONOTONIC,0);
    int milliseconds = atoi(argv[1]);
    struct itimerspec timspec;
    struct sched_param schedparm;

    memset(&schedparm, 0, sizeof(schedparm));
    schedparm.sched_priority = 1; // lowest rt priority
    sched_setscheduler(0, SCHED_FIFO, &schedparm);

    bzero(&timspec, sizeof(timspec));
    timspec.it_interval.tv_sec = 0;
    timspec.it_interval.tv_nsec = milliseconds * 1000000;
    timspec.it_value.tv_sec = 0;
    timspec.it_value.tv_nsec = 1;

    int res = timerfd_settime(timerfd, 0, &timspec, 0);
    if(res < 0){
       perror("timerfd_settime:");
    }
    uint64_t expirations = 0;
    int iterations = 0;
    while( res = read(timerfd, &expirations, sizeof(expirations))){
        if(res < 0){ perror("read:"); continue; }
        if(expirations > 1){
            printf("%ld expirations, %d iterations\n", expirations, iterations);
            break;
        }
        iterations++;
    }
}

if you are using threads you should use pthread_setschedparam instead of sched_setscheduler

realtime also isn't about low latency, it's about guarantees, RT means that if you want to wake up exactly once every second on the second, you WILL, the normal scheduling does not give you this, it might decide to wake you up 100ms later, because it had some other work to do at that time anyway. If you want to wake up every 10ms and you REALLY do need to, then you should set yourself to run as a realtime task then the kernel will wake you up every 10ms without fail. unless a higher priority realtime task is busy doing stuff.

If you need to guarantee that your wakeup interval is exactly some time it doesn't matter if it's 1ms or 1 second, you won't get it unless you run as a realtime task. There are good reasons the kernel will do this to you (saving power is one of them, higher throughput is another, there are others), but it's well within it's rights to do so since you never told it you need better guarantees. Most stuff doesn't actually need to be this accurate, or need to never miss so you should think hard about weather or not you really do need it.

quoting from http://www.ganssle.com/articles/realtime.htm

A hard real time task or system is one where an activity simply must be completed - always - by a specified deadline. The deadline may be a particular time or time interval, or may be the arrival of some event. Hard real time tasks fail, by definition, if they miss such a deadline.

Notice this definition makes no assumptions about the frequency or period of the tasks. A microsecond or a week - if missing the deadline induces failure, then the task has hard real time requirements.

Soft realtime is pretty much the same, except that missing a deadline, while undesirable, is not the end of the world (for example video and audio playback are soft realtime tasks, you don't want to miss displaying a frame, or run out of buffer, but if you do it's just a momentary hiccough, and you simply continue). If what you are trying to do is 'soft' realtime I wouldn't bother with running at realtime priority since you should generally get your wakeups in time (or at least close to it).

EDIT:

If you aren't running realtime the kernel will by default give any timers you make some 'slack' so that it can merge your request to wake up with other events that happen at times close to the one you asked for (that is if the other event is within your 'slack' time it will not wake you at the time you asked, but a little earlier or later, at the same time it was already going to do something else, this saves power)

For a little more info see High- (but not too high-) resolution timeouts and Timer slack (note I'm not sure if either of those things is exactly what's really in the kernel since both those articles are about lkml mailing list discussions, but something like the first one really is in the kernel

Spudd86
+1  A: 

I've got a feeling that your test is very hardware dependent. When I ran your sample program on my system, it appeared to hang at 1ms. To make your test at all meaningful on my computer, I had to change from milliseconds to microseconds. (I changed the multiplier from 1_000_000 to 1_000.)

$ grep 1000 test.c
    timspec.it_interval.tv_nsec = microseconds * 1000;
$ for i in 1 2 4 5 7 8 9 15 16 17\
 31 32 33 47 48 49 63 64 65 ; do\
 echo "intervals of $i microseconds";\
 ./test $i;done
intervals of 1 microseconds
11 expirations, 0 iterations
intervals of 2 microseconds
5 expirations, 0 iterations
intervals of 4 microseconds
3 expirations, 0 iterations
intervals of 5 microseconds
2 expirations, 0 iterations
intervals of 7 microseconds
2 expirations, 0 iterations
intervals of 8 microseconds
2 expirations, 0 iterations
intervals of 9 microseconds
2 expirations, 0 iterations
intervals of 15 microseconds
2 expirations, 7788 iterations
intervals of 16 microseconds
4 expirations, 1646767 iterations
intervals of 17 microseconds
2 expirations, 597 iterations
intervals of 31 microseconds
2 expirations, 370969 iterations
intervals of 32 microseconds
2 expirations, 163167 iterations
intervals of 33 microseconds
2 expirations, 3267 iterations
intervals of 47 microseconds
2 expirations, 1913584 iterations
intervals of 48 microseconds
2 expirations, 31 iterations
intervals of 49 microseconds
2 expirations, 17852 iterations
intervals of 63 microseconds
2 expirations, 24 iterations
intervals of 64 microseconds
2 expirations, 2888 iterations
intervals of 65 microseconds
2 expirations, 37668 iterations

(Somewhat interesting that I got the longest runs from 16 and 47 microseconds, but 17 and 48 were awful.)

time(7) has some suggestions on why our platforms are so different:

   High-Resolution Timers
       Before Linux 2.6.21, the accuracy of timer and sleep system
       calls (see below) was also limited by the size of the jiffy.

       Since Linux 2.6.21, Linux supports high-resolution timers
       (HRTs), optionally configurable via CONFIG_HIGH_RES_TIMERS.  On
       a system that supports HRTs, the accuracy of sleep and timer
       system calls is no longer constrained by the jiffy, but instead
       can be as accurate as the hardware allows (microsecond accuracy
       is typical of modern hardware).  You can determine whether
       high-resolution timers are supported by checking the resolution
       returned by a call to clock_getres(2) or looking at the
       "resolution" entries in /proc/timer_list.

       HRTs are not supported on all hardware architectures.  (Support
       is provided on x86, arm, and powerpc, among others.)

All the 'resolution' lines in my /proc/timer_list are 1ns on my (admittedly ridiculously powerful) x86_64 system.

I decided to try to figure out where the 'breaking point' is on my computer, but gave up on the 110 microsecond run:

$ for i in 70 80 90 100 110 120 130\
 ; do echo "intervals of $i microseconds";\
 ./test $i;done
intervals of 70 microseconds
2 expirations, 639236 iterations
intervals of 80 microseconds
2 expirations, 150304 iterations
intervals of 90 microseconds
4 expirations, 3368248 iterations
intervals of 100 microseconds
4 expirations, 1964857 iterations
intervals of 110 microseconds
^C

90 microseconds ran for three million iterations before it failed a few times; that's 22 times better resolution than your very first test, so I'd say that given the right hardware, 10ms shouldn't be anywhere near difficult. (90 microseconds is 111 times better resolution than 10 milliseconds.)

But if your hardware doesn't provide the timers for high resolution timers, then Linux can't help you without resorting to SCHED_RR or SCHED_FIFO. And even then, perhaps another kernel could better provide you with the software timer support you need.

Good luck. :)

sarnold
yes the kernel has hrtimers so it CAN wake you up within 1ns of when you asked it to, weather or not it WILL is a different story, timers have some 'slack' (if you aren't a realtime task) so that the kernel can reduce the number wakeups in total. (ie if it already has something to do within the slack period it will make you wake up the same time as whatever else see http://lwn.net/Articles/296578/)
Spudd86