Data Execution Prevention (DEP) is a security feature included in modern Microsoft Windows operating systems that is intended to prevent an application or service from executing code from a non-executable memory region. This helps prevent certain exploits that store code via a buffer overflow, for example. DEP runs in two modes: hardware-enforced DEP for CPUs that can mark memory pages as nonexecutable, and software-enforced DEP with a limited prevention for CPUs that do not have hardware support. Software-enforced DEP does not protect from execution of code in data pages, but instead from another type of attack (SEH overwrite).
DEP was introduced in Windows XP Service Pack 2 and is included in Windows XP Tablet PC Edition 2005, Windows Server 2003 Service Pack 1 and later, Windows Vista, and Windows Server 2008.
Hardware-enforced DEP enables the NX bit
on compatible CPUs, through the automatic use of PAE kernel
in 32-bit Windows and the native support on 64-bit kernels. Windows Vista
DEP works by marking certain parts of memory as being intended to hold only data, which the NX or XD bit enabled processor then understands as non-executable. This helps prevent buffer overflow
attacks from succeeding. In Windows Vista, the DEP status for a process, that is, whether DEP is enabled or disabled for a particular process can be viewed on the Processes
tab in the Windows Task Manager
In some instances, Data Execution Prevention can have the unintended consequence of preventing legitimate software from executing. In these cases, the affected software needs to be flagged as being allowed to execute code in those parts of memory, but this itself leads to a possible attack if the application isn't rigorous in validating data that is passed into a region of memory that is marked as being executable.
If the x86 processor supports this feature in hardware, then the NX features are turned on automatically in Windows by default. If the feature is not supported by the x86 processor, then no protection is given. Outside of the x86 architecture, a version of NX also exists for Intel's IA-64 architecture that is supported by Windows.
Software DEP, while unrelated to the NX bit, is what Microsoft calls their enforcement of "Safe Structured Exception Handling". Software DEP/SafeSEH simply checks when an exception is thrown to make sure that the exception is registered in a function table for the application, and requires the program to be built with it. However, even though it creates an impression that software DEP is related to the prevention of executing code in data pages, it is a different form of protection.
Unlike similar protection schemes available on other operating systems, DEP provides no address space layout randomization
(ASLR, a feature now available in Windows Vista
), which may allow return-to-libc attacks
that could feasibly be used to disable DEP during an attack
The possibility has now been demonstrated against Windows Hardware-enforced DEP by authors "skape & Skywing" in the following Uninformed
which relies on a return-to-libc style attack. This technique relies on directly pointing the EIP register to the known service-pack-dependent location which applies the OptIn/OptOut mechanism. It is reliant on the boot-time option of OptOut/OptIn being available. If all pages are strictly enforced, then this attack will not succeed. The PaX
documentation further elaborates
on why ASLR is necessary.
DEP is occasionally the cause of software problems, usually with older software. However, it also fixes problems with some applications, like the Age of Mythology game.
Users have experienced problems using various command line commands that are a part of Microsoft's Services for Unix, which is included as part of Vista as well as Windows Server 2003 Sevice Pack 2.
In most cases, these problems may be solved by disabling the DEP features. DEP can be turned off on a per-application basis, or turned off entirely for all non-essential Windows programs and services. Microsoft recommends that DEP not be globally disabled where an application crashes due to a DEP error. Instead, the author or vendor of the offending software should firstly be contacted for an updated version that does not violate DEP, or failing that, that DEP be disabled on an exception basis for the offending application only.
The "COM surrogate has stopped working" error that is sometimes received in Windows Vista while viewing media folders can be corrected by turning off DEP for the dllhost.exe. This error surfaces in Windows Vista because many third party video and audio codecs are not DEP-compliant. This problem may also generally be resolved by updating any installed codecs to their latest versions.
DEP problems can occasionally occur with DEP-compliant core operating system components such as Windows Explorer, Internet Explorer and Windows Installer as they support in-process third party extensions or plugins that may not be DEP-compliant.
DEP configuration for the system is controlled through switches in the Boot.ini file, while opt-in or opt-out settings can be configured by using the System dialog box in Control Panel
The Boot.ini file settings are as follows:
Note policy_level is defined as AlwaysOn, AlwaysOff, OptIn, or OptOut.
- OptIn: This setting is the default configuration for Windows XP. On systems with processors that can implement hardware-enforced DEP, DEP is enabled by default for limited system binaries and programs that "opt-in." With this option, only Windows system binaries are covered by DEP by default.
- OptOut: This setting is the default configuration for Windows 2003 SP1. DEP is enabled by default for all processes. A list of specific programs that should not have DEP applied can be entered using the System dialog box in Control Panel. Network administrators can use the Application Compatibility Toolkit to "opt-out" one or more programs from DEP protection. System compatibility fixes, or shims, for DEP do take effect. Also note that Windows silently disables DEP for certain executables, such as those packaged with ASPack.
- AlwaysOn: This setting provides full DEP coverage for the whole system. All processes always run with DEP applied. The exceptions list to exempt specific programs from DEP protection is not available. System compatibility fixes for DEP do not take effect. Programs that have been opted-out by using the Application Compatibility Toolkit run with DEP applied.
- AlwaysOff: This setting does not provide any DEP coverage for any part of the system, regardless of hardware DEP support. (except in Windows Vista Ultimate)