An uninterruptable process is a process which happens to be in a system call (kernel function) that cannot be interrupted by a signal.
To understand what that means, you need to understand the concept of an interruptable system call. The classic example is read(). This is a system call that can take a long time (seconds) since it can potentially involve spinning up a hard drive, or moving heads. During most of this time, the process will be sleeping, blocking on the hardware.
While the process is sleeping in the system call, it can receive a unix asynchronous signal (say, SIGTERM), then the following happens:
- The system calls exits prematurely, and is set up to return -EAGAIN to userspace.
- The signal handler is executed.
- If the process is still running, it gets the return value from the system call, and if it is written correctly it will make the same call again.
The crux of the issue is that (for some reason I do not really understand), the execution needs to get out of the system call for the userspace signal handler to run.
On the other hand, some system calls are not allowed to be interrupted in this way. If the system calls stalls for some reason, the process can indefinitely remains in this unkillable state.
LWN ran a nice article that touched this topic in July.
To answer the original question:
How to prevent this from happening: figure out which driver is causing you trouble, and either stop using, or become a kernel hacker and fix it.
How to kill an uninterruptible process without rebooting: somehow make the system call terminate. Frequently the most effective manner to do this without hitting the power switch is to pull the power chord. You can also become a kernel hacker and make the driver use TASK_KILLABLE, as explained in the LWN article.