It is actually very simple. Because Windows is a multitasking operating system, it continually switches (each X milliseconds) from one application to the next. By giving each program very often a very short time to run, it creates the illusion that the programs are working simultaneously.
When an application hangs, the application is probably in a long (possibly endless) loop. Windows keeps giving the application a short time to run and doesn't notice this (unless you want to interact with the application and it doesn't respond within a second). This is the first type of 'crash'.
In the second type, a real crash, some serious error occurred so that Windows cannot let the program continue. For example, the program attempts to write to a memory area that is reserved for some other program or Windows itself. The processor has a build-in mechanism which generates an interrupt (sort of event for the processor) when this happens. Windows is programmed to react on this interrupt, and because it has no way to fix the problem, it will treat the program as being 'crashed' and will terminate it immediately.
As mentioned, writing to the wrong memory address causes an (protection) interrupt by the processor automatically. Other things which may cause such an interrupt for an unrecoverable error are amongst others:
- Reading from an unallowed memory address
- Insufficient memory for this specific application (however, paging mostly removes this problem)
- Attempt to execute unexecutable memory (for example, data)
- Jumping to an invalid address (e.g. in the middle of a machine instruction)
Windows constructs special tables which are used by the Memory Management Unit (MMU) on the processor, which contains information about which areas of the memory the current process can access. For each process, this table is different. Obviously, because each process resides at a different location in memory, and it has to be able to access its own data and code.
So the OS using special access tables, combined with protection interrupts fired by the processor, are mainly the reason that a program doesn't take the whole operating system with it. Otherwise, timesharing allows the rest of the OS and programs to continue when a program is hanging.