How does Windows protect transition into kernel mode

Question

How does Windows protect against a user-mode thread from arbitrarily transitioning the cpu to kernel-mode?

I understand these things are true:

1. User-mode threads DO actually transition to kernel-mode when a system call is made through NTDLL.
2. The transition to kernel-mode is done through processor-specific instructions.

So what is special about these system calls through NTDLL? Why can't the user-mode thread fake-it and execute the processor-specific instructions to transition to kernel-mode? I know I'm missing some key piece of Windows architecture here...what is it?

Answer 1

Intel CPUs enforce security using what's called 'Protection Rings'.

There are 4 of these, numbered from 0 to 3. Code running in ring 0 has the highest privileges; it can (practically) do whatever it pleases with your computer. The code in ring 3, on the other hand, is always on a tight leash; it has only limited powers to influence things. And rings 1 and 2 are currently not used for any purpose at all.

A thread running in a higher privileged ring (such as 0) can transition to lower privilege ring (such as 0) at will. However, the transition the other way around is strictly regulated. This is how the security of high privileged resources (such as memory) etc. is maintained.

Naturally, your user mode code (applications and all) runs in ring 3 while the OS's code runs in ring 0. This ensures that the user mode threads can't mess with the OS's data structures and other critical resources.

For details on how all this is actually implemented you could read this article. In addition, you may also want to go through Intel Manuals, especially Vol 1 and Vol 3A, which you can download here.

This is the story for Intel processors. I'm sure other architectures have something similar going on.

Answer 2

I think (I may be wrong) that the mechanism which it uses for transition is simple:

• User-mode code executes a software interrupt
• This (interrupt) causes a branch to a location specified in the interrupt descriptor table (IDT)

The thing that prevents user-mode code from usurping this is as follows: you need to be priviledged to write to the IDT; so only the kernel is able to specify what happens when an interrupt is executed.

Answer 3

You're probably thinking the user more thread is calling into Ring 0, but that's not what's actually happening. The user mode thread is causing an exception that's caught by the Ring 0 code. The user mode thread is halted and the CPU switches to a kernel/ring 0 thread, which can then inspect the context (e.g., call stack & registers) of the user mode thread to figure out what to do. Before syscall, it really was an exception rather than a special exception specifically to invoke ring 0 code.

If you take the advice of the other responses and read the intel manuals, you'll see syscall/sysenter don't take any parameters - the O/S decides what happens. You can't call arbitrary code. WinNT uses function #'s that map to which kernel mode function the user mode code will execute (e.g. NtOpenFile is fnc 75h on my XP machine (the #'s change all the time, that's one of the jobs of NTDll is to map a function call to a fnc#, put it in EAX, point EDX to the incoming parameters then invoke sysenter).

Answer 4

Code running in User Mode (Ring 3) can't arbitrarily change to Kernel Mode (Ring 0). It can only do so using special routes -- jump gates, interrupts, and sysenter vectors. These routes are highly protected and input is scrubbed so that bad data can't (shouldn't) cause bad behavior.

All of this is set up by the kernel, usually on startup. It can only be configured in Kernel Mode so User-Mode code can't modify it.

Answer 5

It's probably fair to say that it does it in a (relatively) similar way to what Linux does. In both cases it's going to be CPU-specific, but on x86 probably either a software interrupt with the INT instruction, or via SYSENTER instruction.

The advantage of looking at how Linux does it is that you can do so without a Windows source licence.

The userspace source part is here here at LXR and the kernel space bit - look at entry_32.S and entry_64.S

Under Linux on x86 there are three different mechanisms, int 0x80, syscall and sysenter.

A library which is built at runtime by the kernel called vdso is called by the C library to implement the syscall function, which uses a different mechanism depending on the CPU and which system call it is. The kernel then has handlers for those mechanisms (if they exist on the specific CPU variant).