Hello,
Is there anyone here with experience in the linux thread scheduler running multithreading applications in the new Quad core processors?. If there is someone like that can you please write here your experience about how is the performance of the kernel managing the different threads, Have you experienced any thread starving or the starving of one of the cores?.
Thank you.
Linux supports using many processors very well itself. If I remember correctly with SMP, Linux supports 4096-processors. What will really make a difference is whether your applications are written to take advantage of multiple processors.
It works very well on a twin quad-core system (V8) we have in production... bloody fast.
But be very wary of Linux's propensity for thread starvation when locks (mutexes) are heavily contested. Imagine a scenario when 10 threads are working with one lock where the lock is needed very frequently but for very short amounts of time and the work done outside of the lock at any given point is less than a timeslice. Linux will very much tend to deliver the lock almost always to one thread to the exclusion of all the others.
This depends on the specific threading package bound into the kernal, too - I believe there are several.
I have gotten absolutely stunning results on our Intel Q6600's, both for parallel make and for some other parallel applications, but I have been careful to avoid excessive parallelism: I generally fork between four and eight threads, so there's not too much contention. If you fork enough threads, you will have noticeable overhead, especially if they contend for the same semaphores. My guess is that thousands of threads are probably too many, and dozens of threads are probably OK. But that's just a guess; if you want to know, you'll have to find someone who has measured, or you'll have to do an experiment yourself.
But for a dozen threads our results have been fabulous.
Given that kernel developers like Christoph Lameter (and Ingo Molnar on the scheduler) have tuned the kernel to work well on 4096 processors, and given the amount of optimizations invested by Intel itself in the issue, with multicore specific tuning both for performance and energy saving, I bet the kernel is by far more optimized than anything any of us can write in userspace.
Same about the threading library; there is currently only one thread library, NPTL for Linux 2.6. LinuxThreads was removed from glibc in the 2.4 release, and NPTL was produced before the 2.6 release. And it's really fast.
Just make sure to avoid using an old kernel, the last release of your distro, or kernel.org, is the best. Before deploying in production, make sure to measure the performance difference, and consider whether that is worth the additional support costs (if any).