How does the NX flag work?

Question

Could you please explain what the NX flag is and how it works (please be technical)?

Answer 1

From Wikipedia

The NX bit, which stands for No eXecute, is a technology used in CPUs to segregate areas of memory for use by either storage of processor instructions (or code) or for storage of data, a feature normally only found in Harvard architecture processors. However, the NX bit is being increasingly used in conventional von Neumann architecture processors, for security reasons.

An operating system with support for the NX bit may mark certain areas of memory as non-executable. The processor will then refuse to execute any code residing in these areas of memory. The general technique, known as executable space protection, is used to prevent certain types of malicious software from taking over computers by inserting their code into another program's data storage area and running their own code from within this section; this is known as a buffer overflow attack.

Answer 2

It marks a memory page non-executable in the virtual memory system and in the TLB (a structure used by the CPU for resolving virtual memory mappings). If any program code is going to be executed from such page, the CPU will fault and transfer control to the operating system for error handling.

Programs normally have their binary code and static data in a read-only memory section and if they ever try to write there, the CPU will fault and then the operating-system normally kills the application (this is known as segmentation fault or access violation).

For security reasons, the read/write data memory of a program is usually NX-protected by default. This prevents an attacker from supplying some application his malicious code as data, making the application write that to its data area and then having that code executed somehow, usually by a buffer overflow/underflow vulnerability in the application, overwriting the return address of a function in stack with the location of the malicious code in the data area.

Some legitimate applications (most notably high-performance emulators and JIT compilers) also need to execute their data, as they compile the code at runtime, but they specifically allocate memory with no NX flag set for that.

Answer 3

Have a look at this 'DEP' found on wikipedia which uses the NX bit. As for supplying the technical answer, sorry, I do not know enough about this but to quote:

Data Execution Prevention (DEP) is a security feature included in modern 
Microsoft Windows operating systems that is intended to prevent an 
application or service from executing code from a non-executable memory region.
....
DEP was introduced in Windows XP Service Pack 2 and is included in Windows XP 
Tablet PC Edition 2005, Windows Server 2003 Service Pack 1 and later, Windows 
Vista, and Windows Server 2008, and all newer versions of Windows.
...
Hardware-enforced DEP enables the NX bit on compatible CPUs, through the 
automatic use of PAE kernel in 32-bit Windows and the native support on 64-bit 
kernels. 
Windows Vista DEP works by marking certain parts of memory as being intended to 
hold only data, which the NX or XD bit enabled processor then understands as 
non-executable.
This helps prevent buffer overflow attacks from succeeding. In Windows Vista, 
the DEP status for a process, that is, whether DEP is enabled or disabled for a 
particular process can be viewed on the Processes tab in the Windows Task 
Manager.

See also here from the MSDN's knowledge base about DEP. There is a very detailed explanation here on how this works.

Hope this helps, Best regards, Tom.