In Windows, how can one create a child process and capture its stdin, stdout, and stderr, without duplicating any inheritable handles?

Question

There are at least three parts to this problem, so bear with me:

1) CreateProcess has a parameter bInheritHandles, that causes the child process to inherit all of the inheritable handles in the parent process. This option must be set to TRUE to allow the parent to specify stdin, stdout, and stderr handles for the child in the STARTUPINFO parameter.

2) In Win32 deleting and renaming files can fail when there is more than one handle open to the same file.

3) The Microsoft CRT's open() function will by default create inheritable handles. Additionally the file handles created by default suffer from problem 2 above.

This magic combination creates the following operational problem: Library A calls open() and doesn't expect subsequent renames and deletes to fail. Elsewhere in the process another library B is calling CreateProcess with bInheritHandles set to TRUE (to capture stdin/out/err) temporarily creating duplicate handles. Now occasionally library A's file operations fail. Naturally library A and B are maintained by separate people. I also know of another library A' that uses open() and suffers from a similar problem.

This kb article discusses a related problem and solution. However it still relies on calling CreateProcess with bInheritHandles set to TRUE in the parent process, so it doesn't solve this problem.

I am wondering if others have hit this problem and if there isn't a well known solution?

The kb article above essentially implies that calling CreateProcess with bInheritHandles set to TRUE is racy, so my inclination is to fix library B such that it never does that. I would do this by:

1. Create a suspended intermediate process (ideally by using rundll to run a custom entry point in library B) with bInheritHandles set to FALSE.
2. Create stdin/out/err pipes and dup the correct ends of those to the intermediate process.
3. Pass the duped handles to the intermediate process somehow.
4. Resume the intermediate process.
5. From the intermediate process fill out the STARTUPINFO with the pipes from the parent and call CreateProcess with bInheritHandles set to TRUE.

Is this a good strategy or is there some better solution? How would you recommend passing the duped handles to the intermediate process in step 3? Is rundll + custom entry point a reliable way to setup the intermediate process in step 1?

Answer 1

If you have access to the actual file handles, you can use SetHandleInformation() to remove the HANDLE_FLAG_INHERIT flag before calling CreateProcess().

Answer 2

You can use the ZwQuerySystemInformation(SystemHandleInformation, ...) ntdll.dll function to find all the handles owned by your process, then all SetHandleInformation on each one, as suggested by Remy, to remove the HANDLE_FLAG_INHERIT flag.