Reading Unicode from redirected STDOUT (C++, Win32 API, Qt)

Question

Hello,

I have a C++ application which dynamically loads plug-in DLLs. The DLL sends text output via std::cout and std::wcout. Qt-based UI must grab all text output from DLLs and display it. The approach with stream buffer replacement doesn't fully work since DLLs might have different instances of cout/wcout due to run-time libraries differences. Thus I have applied Windows-specific STDOUT redirection as follows:

StreamReader::StreamReader(QObject *parent) :
    QThread(parent)
{
    // void
}

void StreamReader::cleanUp()
{
    // restore stdout
    SetStdHandle (STD_OUTPUT_HANDLE, oldStdoutHandle);

    CloseHandle(stdoutRead);
    CloseHandle(stdoutWrite);
    CloseHandle (oldStdoutHandle);

    hConHandle = -1;

    initDone = false;
}

bool StreamReader::setUp()
{

    if (initDone)
    {
        if (this->isRunning())
            return true;
        else
            cleanUp();
    }

    do
    {
        // save stdout
        oldStdoutHandle = ::GetStdHandle (STD_OUTPUT_HANDLE);

        if (INVALID_HANDLE_VALUE == oldStdoutHandle)
            break;

        if (0 == ::CreatePipe(&stdoutRead, &stdoutWrite, NULL, 0))
            break;

        // redirect stdout, stdout now writes into the pipe
        if (0 == ::SetStdHandle(STD_OUTPUT_HANDLE, stdoutWrite))
            break;

        // new stdout handle
        HANDLE lStdHandle = ::GetStdHandle(STD_OUTPUT_HANDLE);

        if (INVALID_HANDLE_VALUE == lStdHandle)
            break;

        hConHandle = ::_open_osfhandle((intptr_t)lStdHandle, _O_TEXT);
        FILE *fp = ::_fdopen(hConHandle, "w");

        if (!fp)
            break;

        // replace stdout with pipe file handle
        *stdout = *fp;

        // unbuffered stdout
        ::setvbuf(stdout, NULL, _IONBF, 0);

        hConHandle = ::_open_osfhandle((intptr_t)stdoutRead, _O_TEXT);

        if (-1 == hConHandle)
            break;

        return initDone = true;

    } while(false);


    cleanUp();

    return false;
}

void StreamReader::run()
{
    if (!initDone)
    {
        qCritical("Stream reader is not initialized!");
        return;
    }

    qDebug() << "Stream reader thread is running...";

    QString s;
    DWORD nofRead  = 0;
    DWORD nofAvail = 0;

    char buf[BUFFER_SIZE+2] = {0};

    for(;;)
    {
        PeekNamedPipe(stdoutRead, buf, BUFFER_SIZE, &nofRead, &nofAvail, NULL);

        if (nofRead)
        {
            if (nofAvail >= BUFFER_SIZE)
            {
                while (nofRead >= BUFFER_SIZE)
                {
                    memset(buf, 0, BUFFER_SIZE);
                    if (ReadFile(stdoutRead, buf, BUFFER_SIZE, &nofRead, NULL)
                        && nofRead)
                    {
                        s.append(buf);
                    }
                }
            }
            else
            {
                memset(buf, 0, BUFFER_SIZE);
                if (ReadFile(stdoutRead, buf, BUFFER_SIZE, &nofRead, NULL)
                    && nofRead)
                {
                    s.append(buf);
                }

            }

            // Since textReady must emit only complete lines,
            // watch for LFs
            if (s.endsWith('\n')) // may be emmitted
            {
                emit textReady(s.left(s.size()-2));
                s.clear();
            }
            else    // last line is incomplete, hold emitting
            {
                if (-1 != s.lastIndexOf('\n'))
                {
                    emit textReady(s.left(s.lastIndexOf('\n')-1));
                    s = s.mid(s.lastIndexOf('\n')+1);
                }
            }

            memset(buf, 0, BUFFER_SIZE);
        }
    }

    // clean up on thread finish
    cleanUp();
}

However, this solution appears to have an obstacle - C runtime library, which is locale-dependent. Thus any output sent to wcout isn't reaching my buffer because C runtime truncates strings at non-printable ASCII characters present in UTF-16 encoded strings. Calling setlocale() demonstrates, that C runtime does string re/encoding. setlocale() is no help for me for very reason that there is no knowledge of the language or locale of the text, since plug-in DLLs read from outside the system and there might be different languages mixed. After giving an N-thought I have decided to drop this solution and revert to cout/wcout buffer replacement and putting requirement for DLLs to call initialization method due to both reasons: UTF16 not passing to my buffer, and then the problem of figuring out encoding in the buffer. However, I am still curious of whether there is a way to get UTF-16 strings through C runtime into pipe 'as is', without locale-dependent conversion?

p.s. any suggestions on cout/wcout redirection to UI rather than the two mentioned approaches are welcome as well :)

Thank you in advance!

Answer 1

I don't know if this is possible, but maybe you could start the DLL in a separate process and capture the output of that process with the Windows equivalent of pipe (whatever that is, but Qt's QProcess should take care of that for you). This would be similar to how Firefox does out of process plugins (default in 3.6.6, but it's been done for a while with 64 bit Firefox and the 32 bit Flash plugin). You'd have to come up with some way to communicate with the DLL in the separate process, like shared memory, but it should be possible. Not necessarily pretty, but possible.

Answer 2

Try:

std::wcout.imbue(std::locale("en_US.UTF-8"));

This is stream-specific, and better than using the global C library setlocale().

However, you may have to tweak the locale name to suit what your runtime supports.

Answer 3

The problem here is that the code conversion from wchar_t to char is being done entirely inside the plug-in DLL, by whatever cout/wcout implementation it happens to be using (which as you say may not be the same as the one that the main application is using). So the only way to get it to behave differently is to intercept that mechanism somehow, such as with streambuf replacement.

However, as you imply, any code you write in the main application isn't necessarily going to be compatible with the library implementation that the DLL uses. For example, if you implement a stream buffer in the main application, it won't necessarily be using the same ABI as the stream buffers in the DLL. So this is risky.

I suggest you implement a wrapper DLL that uses the same C++ library version as the plug-in, so it's guaranteed to be compatible, and in this wrapper DLL do the necessary intervention in cout/wcout. It could load the plug-in dynamically, and so could be reusable with any plug-in that uses that library version. Alternatively, you could create some reusable source code that could be compiled specifically for each plug-in, thus producing a sanitized version of each plug-in.

Once the DLL is wrapped, you can substitute a stream buffer into cout/wcout that saves the data to memory, as I think you were originally planning, and not have to mess with file handles at all.

PS: If you ever do need to make a wstream that converts to and from UTF-8, then I recommend using Boost's utf8_codecvt_facet as a very neat way of doing it. It's easy to use, and the documentation has example code. (In this scenario you would have to compile a version of Boost specifically for the library version that the plug-in is using, but not in the general case.)