What is __kernel_vsyscall?

Question

I got a core that looks very different from the ones I usually get - most of the threads are in __kernel_vsyscall() :

  9 process 11334  0xffffe410 in __kernel_vsyscall ()
  8 process 11453  0xffffe410 in __kernel_vsyscall ()
  7 process 11454  0xffffe410 in __kernel_vsyscall ()
  6 process 11455  0xffffe410 in __kernel_vsyscall ()
  5 process 11474  0xffffe410 in __kernel_vsyscall ()
  4 process 11475  0xffffe410 in __kernel_vsyscall ()
  3 process 11476  0xffffe410 in __kernel_vsyscall ()
  2 process 11477  0xffffe410 in __kernel_vsyscall ()
  1 process 11323  0x08220782 in MyClass::myfunc ()

What does that mean?

EDIT: In particular, I usually see a lot of threads in "pthread_cond_wait" and "___newselect_nocancel" and now those are on the second frame in each thread - why is this core different?

Answer 1

When you make a system call (like reading from a file, talking to hardware, writing to sockets) you're actually creating an interrupt. The system then handles the interrupt in kernel mode and your call returns with the result. Most of the time it's unusual for you to have a lot of threads in syscall unless you're making blocking calls, in which case it's expected.

More specifically, it means the thread is waiting on a kernel level system call. But that's (unfortunately for my points) already in the name :)

Answer 2

__kernel_vsyscal is the method used by linux-gate.so (a part of the Linux kernel) to make a system call using the fastest available method, preferably the sysenter instruction. The thing is properly explained by Johan Petersson.

Answer 3

In addition to the already given good link to explanation of what linux-gate.so is, I'd like to answer "why is this core different?". Most recent (newer than 2.5.68) 32-bit Linux systems use VDSO page (aka linux-gate.so.1), and 64-bit systems will soon start as well (64-bit VDSO was introduced in kernel 2.6.24).

If you develop on an older system, or with an old glibc, then you would never see __kernel_vsyscall(), either because the kernel didn't create VDSO at all, or because (old) glibc doesn't use it even when VDSO is present.

Answer 4

As Adam said, the main reason is performance. See this link for some old numbers http://lkml.org/lkml/2002/12/9/13.

If you have a vDSO enabled kernel, you're not using interrupts to run syscalls, as Stefan said, actually was because interrupts was getting slower that the whole vDSO thing was added to the kernel.