16-Bit Apps running under NTVDM

Answer 1

From this link about VDMDBG functions, you may be able to P/Invoke "VDMEnumProcessWOW()", then enumerate modules within the process using PSAPI.

Note Regarding 16-bit DOS Applications:

None of the VDMDBG functions work with 16-bit DOS applications. To enumerate DOS VDMs, you need to use another method. First, you could use VDMEnumProcessWOW() to make a list of all Win16 VDMs, and then enumerate all instances of NTVDM.exe using some other scheme (such as PSAPI). Any NTVDM.exe from the full enumeration that was not in the Win16 list is a DOS VDM. You can create and terminate 16-bit DOS applications with CreateProcess() and TerminateProcess().

Hope that helps...

Answer 2

This works for me:

• Follow the instructions at Description of the Software Restriction Policies in Windows XP to open the local or domain policy editor.

• Under Software Restriction Policies -> Additional Rules, right click and select New Hash Rule.

• Browse to (for example) edit.com. Make sure Security Level is set to Disallowed. Click OK.

Now,

C:\>edit
The system cannot execute the specified program.

(I get the same results from command.com and cmd.exe -- under Win XP)

Answer 3

The trick is not to use VDMEnumProcessWOW (which gives the VDMs), but to use VDMEnumTasksWOW. The enumerator function that you pass to this function will be called for each 16 bit task in the specified VDM.

I haven't checked it myself, but according to the documentation, this library of CodeProject does exactly that, if you pass in the PROC16 enum value. It's C++, if you need help compiling that code and calling it from C#, let me know and I'll give you an example.

A program that uses this technique is Process Master, it comes with full source. I suggest you run it to find out whether it gives the info you need, and if so, you can apply this method to your own application (it doesn't run on Windows Vista or 7, it uses old VB5 code, apparently it's not compatible. It should run on XP).

If things with these functions do not go as planned, you may be on Vista and may need the hotfix described in this StackOverflow question, which points to downloading a hotfix, which is in turn described here:

"An application that uses the VDMEnumProcessWOW function to enumerate virtual DOS machines returns no output or incorrect output on a computer that is running a 32-bit version of Windows Vista"

Update: while this seems promising, I applied the patch, ran several versions of the code, including Microsoft's, and while they all work on XP, they fail silently (no error, or wrong return value) on Vista.

---

The "kinda" working code

Update: I experimented with (amongst others) with the following code, which compiles fine in C# (and can be written simpler, but I didn't want to run a marshal-mistake risk). When you add these functions, you can call Enum16BitProcesses, which will write the filenames of the EXE files of the 16 bit processes to the Console.

I can't run it on Vista 32 bit. But perhaps others can try and compile it, or find the error in the code. It would be nice to know whether it works on other systems:

public class YourEnumerateClass
{
    public static void Enum16BitProcesses()
    {
        // create a delegate for the callback function
        ProcessTasksExDelegate procTasksDlgt = 
             new ProcessTasksExDelegate(YourEnumerateClass.ProcessTasksEx);

        // this part is the easy way of getting NTVDM procs
        foreach (var ntvdm in Process.GetProcessesByName("ntvdm"))
        {
            Console.WriteLine("ntvdm id = {0}", ntvdm.Id);
            int apiRet = VDMEnumTaskWOWEx(ntvdm.Id, procTasksDlgt, IntPtr.Zero);
            Console.WriteLine("EnumTaskWOW returns {0}", apiRet);
        }

    }

    // declaration of API function callback
    public delegate bool ProcessTasksExDelegate(
        int ThreadId,
        IntPtr hMod16,
        IntPtr hTask16,
        IntPtr ptrModName,
        IntPtr ptrFileName,
        IntPtr UserDefined
        );

    // the actual function that fails on Vista so far
    [DllImport("VdmDbg.dll", SetLastError = false, CharSet = CharSet.Auto)]
    public static extern int VDMEnumTaskWOWEx(
        int processId, 
        ProcessTasksExDelegate TaskEnumProc, 
        IntPtr lparam);

    // the actual callback function, on Vista never gets called
    public static bool ProcessTasksEx(
        int ThreadId,
        IntPtr hMod16,
        IntPtr hTask16,
        IntPtr ptrModName,
        IntPtr ptrFileName,
        IntPtr UserDefined
        )
    {
        string filename = Marshal.PtrToStringAuto(ptrFileName);
        Console.WriteLine("Filename of WOW16 process: {0}", filename);
        return false;       // false continues enumeration
    }

}

Update: Intriguing read by the renown Matt Pietrek. Mind the sentence, somewhere near the end:

"For starters, MS-DOS-based programs seem to always run in separate NTVDM sessions. I was never able to get an MS-DOS-based program to run in the same session as a 16-bit Windows-based program. Nor was I able to get two independently started MS-DOS-based programs to run in the same NTVDM session. In fact, NTVDM sessions running MS-DOS programs don't show up in VDMEnumProcessWOW enumerations."

Seems that, to find out what processes are loaded, you'll need to write a hook into NTVDM or write a listener that monitors access to the file. When the application that tries to read a certain DOS file is NTVDM.exe, it's bingo. You may want to write a DLL that's only attached to NTVDM.exe, but now we're getting a bit ahead of ourselves. Long story short: this little ride into NTVDM has shown "possibilities" that appeared real hoaxes in the end.

There's one other way, but time is too short to create an example. You can poke around in the DOS memory segments and the EXE is usually loaded at the same segment. But I'm unsure if that eventually will lead to the same result and whether it's worth the effort.