Can I prevent a Linux user space pthread yielding in critical code?

Question

I am working on an user space app for an embedded Linux project using the 2.6.24.3 kernel. My app passes data between two file nodes by creating 2 pthreads that each sleep until a asynchronous IO operation completes at which point it wakes and runs a completion handler.

The completion handlers need to keep track of how many transfers are pending and maintain a handful of linked lists that one thread will add to and the other will remove.

// sleep here until events arrive or time out expires
for(;;) {
    no_of_events = io_getevents(ctx, 1, num_events, events, &timeout);
    // Process each aio event that has completed or thrown an error
    for (i=0; i<no_of_events; i++) {
        // Get pointer to completion handler
        io_complete = (io_callback_t) events[i].data;
        // Get pointer to data object
        iocb = (struct iocb *) events[i].obj;
        // Call completion handler and pass it the data object
        io_complete(ctx, iocb, events[i].res, events[i].res2);
    }
}

My question is this...

Is there a simple way I can prevent the currently active thread from yielding whilst it runs the completion handler rather than going down the mutex/spin lock route?

Or failing that can Linux be configured to prevent yielding a pthread when a mutex/spin lock is held?

Answer 1

You can use the sched_setscheduler() system call to temporarily set the thread's scheduling policy to SCHED_FIFO, then set it back again. From the sched_setscheduler() man page:

A SCHED_FIFO process runs until either it is blocked by an I/O request, it is preempted by a higher priority process, or it calls sched_yield(2).

(In this context, "process" actually means "thread").

However, this is quite a suspicious requirement. What is the problem you are hoping to solve? If you are just trying to protect your linked list of completion handlers from concurrent access, then an ordinary mutex is the way to go. Have the completion thread lock the mutex, remove the list item, unlock the mutex, then call the completion handler.

Answer 2

I think you'll want to use mutexes/locks to prevent race conditions here. Mutexes are by no way voodoo magic and can even make your code simpler than using arbitrary system-specific features, which you'd need to potentially port across systems. Don't know if the latter is an issue for you, though.

Answer 3

I believe you are trying to outsmart the Linux scheduler here, for the wrong reasons.

The correct solution is to use a mutex to prevent completion handlers from running in parallel. Let the scheduler do its job.