How to grab parent process standard output?

Question

I'm writing an utility (http://reg2run.sf.net) which in case execution without arguments works as windows application (shows OpenFileDialog, etc), otherwise - as console application.

So, in first case I don't want to show a console window, that's why project is Windows Application. But in second - I need to show it, and it's created with

if (ptrNew == IntPtr.Zero)
{
    ptrNew = GetStdHandle(-11);
}
if (!AllocConsole())
{
    throw new ExternalCallException("AllocConsole");
}
ptrNew = CreateFile("CONOUT$", 0x40000000, 2, IntPtr.Zero, 3, 0, IntPtr.Zero);
if (!SetStdHandle(-11, ptrNew))
{
    throw new ExternalCallException("SetStdHandle");
}
StreamWriter newOut = new StreamWriter(Console.OpenStandardOutput());
newOut.AutoFlush = true;
Console.SetOut(newOut);
Console.SetError(newOut);

And what I want - is to grab parent process standard output and use it, if it exists (in case execution via cmd.exe or Far Manager). How can I do it?

I tried

static Process GetParentProc()
{
int pidParent = 0;
int pidCurrent = Process.GetCurrentProcess().Id;

IntPtr hSnapshot = CreateToolhelp32Snapshot(2, 0);
if (hSnapshot == IntPtr.Zero)
{
 return null;
}

PROCESSENTRY32 oProcInfo = new PROCESSENTRY32();
oProcInfo.dwSize = (uint)Marshal.SizeOf(typeof(PROCESSENTRY32));

if (!Process32First(hSnapshot, ref oProcInfo))
{
 return null;
}
do
{
 if (pidCurrent == oProcInfo.th32ProcessID)
 {
  pidParent = (int)oProcInfo.th32ParentProcessID;
 }
}
while (pidParent == 0 && Process32Next(hSnapshot, ref oProcInfo));

if (pidParent > 0)
{
 return Process.GetProcessById(pidParent);
}
else
{
 return null;
}

and

StreamWriter newOut = GetParentProc().StandardInput;

but got InvalidOperationException: StandardIn has not been redirected. Because of

GetParentProc().StartInfo.RedirectStandardOutput = false

Answer 1

There are several approaches for applications that need to choose whether to act as console or GUI applications, depending on context, on Windows:

1. Have two separate applications, and have one conditionally start the other.
2. A variant of the above strategy, have two applications, one called 'app.com' (i.e. just rename a console EXE with COM extension) and the other called 'app.exe', so that command-line invocations will find app.com first. Because of ancient DOS compatibility, .COM executables are found before .EXEs. (This in configurable in Windows; see the PATHEXT environment variable.)
3. The rxvt/Cygwin technique, which is one I haven't really seen documented anywhere else.

Let me go into a little bit of detail about how rxvt on Cygwin works. Rxvt is a terminal emulator that normally runs on the X Window system. Because of the limitations of the Win32 console, Cygwin packages it as a more fully-featured console, with support for things like lots of lines of history, dynamic resizing, per-instance configurable fonts and colour themes, non-application-freezing mouse select and copy, etc. In order to run natively on Windows, rxvt shipped with Cygwin includes a tiny X11 wrapper library for Win32. Rxvt on Windows is actually a console application for compatibility reasons with existing native Win32 executables, but most of the time you never see the console; you just see the rxvt terminal emulator window itself.

The way it works is specifically implemented in rxvt/W11/wrap/wrap.c in the rxvt source tree, in the function called hideConsole(). Basically, it opens up its console (with a CreateFile("CONOUT$" ...)), and checks to see if the cursor position is at (0,0) (using GetConsoleScreenBufferInfo() on the console handle).

If it is, then it infers that it has been started as a standalone application, rather than from a console parent application, and thus it knows the OS has created a dedicated Win32 console for the process. It proceeds to hide this console window, but it has to find it first. It uses SetConsoleTitle to set the console window's caption to a unique value based on the name of the application and the current thread ID. It then uses FindWindow to find this window's handle (periodically Sleeping for a few ms if necessary for the title to change, because the console windows are actually controlled by a different process entirely in Windows). When it eventually finds the window handle, it hides it with ShowWindowAsync, passing in SW_HIDE.

Using this approach, you can write an application that:

• if started from a console parent, it can continue to use this console
• if started as an application, it can optionally choose whether or not to hide the console

The only downside is a very brief flash of a console window at application startup.

Answer 2

You can always the following P/Invoke method:

[DllImport("kernel32.dll")]
static extern bool AttachConsole(int dwProcessId);

const int ATTACH_PARENT_PROCESS = -1;