programmatically check for subsystem

Question

I have a .exe created with a windows subsystem. I copy that .exe to another .exe, and I run:

editbin.exe /SUBSYSTEM:CONSOLE my.exe

So my intention is to have a .exe that runs with a GUI, and another .exe that is meant for command line operations (no GUI).

How do I check what subsystem is currently active in my C++ code?

Answer 1

Subsystem type (GUI, console, etc.) is stored in the PE header, which you can access via the ImageHlp functions. You can get it with the following code:

// Retrieve the header for the exe.  GetModuleHandle(NULL) returns base address
// of exe.
PIMAGE_NT_HEADERS header = ImageNtHeader((PVOID)GetModuleHandle(NULL));
if (header->OptionalHeader.Subsystem == IMAGE_SUBSYSTEM_WINDOWS_CUI)
{
    // Console application.
}

Relevent MSDN entries:

ImageNtHeader IMAGE_NT_HEADERS IMAGE_OPTIONAL_HEADER

Answer 2

Look at the ImageLoad function in the Imagehlp library. This returns a LOADED_IMAGE structure which has a IMAGE_NT_HEADERS structure in its FileHeader field. The OptionalHeader field in this structure is an IMAGE_OPTIONAL_HEADER which has a Subsytem filed containg the information that you want.

Answer 3

Far easier than spelunking in your own headers: check if you've got console handles. For a GUI subsystem app, GetStdHandle() will return NULL handles.