CreatePipe and necessary permissions in C#

Question

Hello,

I'm trying to add a Sokoban solver (written in C++) to my program in C# (I have got a class in my program that handles marshalling of C++ interface). My program loads solver.dll library which loads solver.exe. Solver.dll and Solver.exe communicate via pipes.

The problem is that when I run my program in Visual Studio (debugging) the DLL is loaded all right and the plugin's functions works great. But when I open my program outside of Visual Studio (simple double-click on MyProgram.exe) and when I try to run solving function the error message: "Got Invalid Pipe" and "Failed to create pipe." appear.

I'm quite at loss what might be the problem but my guess is that there's a problem with some permissions.

Here are two parts of code from plugin author:

In solver.dll file:

 //create pipes
 SECURITY_ATTRIBUTES attr;
 attr.nLength=sizeof(attr);
 attr.lpSecurityDescriptor=NULL;
 attr.bInheritHandle=TRUE;
 if(!CreatePipe(&hPipeOutRead,&hPipeOutWrite,&attr,0)
         ||!CreatePipe(&hPipeInRead,&hPipeInWrite,&attr,0) )
 {
         ReportError("Failed to create pipe.");
         return SOKOBAN_PLUGIN_RESULT_FAILURE;
 }

 //set std handle for deliver pipe
 HANDLE hStdInSave, hStdOutSave;
 hStdInSave=GetStdHandle(STD_INPUT_HANDLE);
 hStdOutSave=GetStdHandle(STD_OUTPUT_HANDLE);
 if(hStdInSave==INVALID_HANDLE_VALUE || hStdOutSave==INVALID_HANDLE_VALUE)
 {
         ReportError("Failed to get std handles.");
         return SOKOBAN_PLUGIN_RESULT_FAILURE;
 }
 if(!SetStdHandle(STD_INPUT_HANDLE,hPipeOutRead)
         ||!SetStdHandle(STD_OUTPUT_HANDLE,hPipeInWrite) )
 {
         ReportError("Failed to redirect std handles.");
         return SOKOBAN_PLUGIN_RESULT_FAILURE;
 }

 //create BoxSearch Process
 STARTUPINFO startInfo;
 ZeroMemory(&startInfo,sizeof(startInfo));
 startInfo.cb=sizeof(startInfo);

 PROCESS_INFORMATION processInfo;
 ZeroMemory(&processInfo,sizeof(processInfo));

 char dllDir[MAX_PATH];
 char exeName[MAX_PATH+100];
 GetModuleFileName(ghinstDLL,dllDir,MAX_PATH);
 *(strrchr(dllDir,'\\')+1)=0;
 strcpy(exeName,dllDir);
 strcat(exeName,"Solver.exe");

 char cmdLine[MAX_PATH+100];
 strcpy(cmdLine,"\"");
 strcat(cmdLine,exeName);
 strcat(cmdLine,"\"");
 strcat(cmdLine," ");
 strcat(cmdLine,API_DLLCommandLine);

 BOOL launchSuccess=CreateProcess(exeName,cmdLine,NULL,NULL,TRUE,0,NULL,dllDir,&startInfo,&processInfo);

At the side of Solver.exe:

//get pipes
 hPipeRead=GetStdHandle(STD_INPUT_HANDLE),
 hPipeWrite=GetStdHandle(STD_OUTPUT_HANDLE);
 if(0==hPipeRead || INVALID_HANDLE_VALUE==hPipeRead
         ||0==hPipeWrite|| INVALID_HANDLE_VALUE==hPipeWrite)
 {
         FormattedMessageBox("Got invalid pipe.");
         return;
 }

What might be the problem? Are there any permissions related to loaded DLL (for pipes)?

Thanks!

Answer 1

Yuck, that's fugly. This can only work well if MyProgram.exe is a console mode application. Required to get input/output redirection working. It probably works in the debugger because of the Visual Studio hosting process.

P/Invoking AllocConsole in your program would solve the issue. I'll guess you don't particularly care for that console window, you'd have to rewrite that interface. Using named pipes would solve it. Realistically, it doesn't make any sense to run the solver as a separate process. Take a look at C++/CLI to get ahead.