Multithreaded IOCP Client Issue

Question

I am writing a multithreaded client that uses an IO Completion Port.

I create and connect the socket that has the WSA_FLAG_OVERLAPPED attribute set.

if ((m_socket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)) == INVALID_SOCKET)
{
    throw std::exception("Failed to create socket.");
}

if (WSAConnectByName(m_socket, L"server.com", L"80", &localAddressLength, reinterpret_cast<sockaddr*>(&localAddress), &remoteAddressLength, &remoteAddress, NULL, NULL) == FALSE)
{
    throw std::exception("Failed to connect.");
}

I associate the IO Completion Port with the socket.

if ((m_hIOCP = CreateIoCompletionPort(reinterpret_cast<HANDLE>(m_socket), m_hIOCP, NULL, 8)) == NULL)
{
    throw std::exception("Failed to create IOCP object.");
}

All appears to go well until I try to send some data over the socket.

SocketData* socketData = new SocketData;
socketData->hEvent = 0;

DWORD bytesSent = 0;
if (WSASend(m_socket, socketData->SetBuffer(socketData->GenerateLoginRequestHeader()), 1, &bytesSent, NULL, reinterpret_cast<OVERLAPPED*>(socketData), NULL) == SOCKET_ERROR && WSAGetLastError() != WSA_IO_PENDING)
{
    throw std::exception("Failed to send data.");
}

Instead of returning SOCKET_ERROR with the last error set to WSA_IO_PENDING, WSASend returns immediately.

I need the IO to pend and for it's completion to be handled in my thread function which is also my worker thread.

unsigned int __stdcall MyClass::WorkerThread(void* lpThis)
{

}

I've done this before but I don't know what is going wrong in this case, I'd greatly appreciate any efforts in helping me fix this problem.

Answer 1

It's not a problem unless you make it so.

As long as you're not calling SetFileCompletionNotificationModes() and setting the flag to skip completion port processing on success then even if WSARecv (or whatever) returns SUCCESS an IO Completion Packet is queued to the IOCP the same as if ERROR_IO_PENDING was returned. Thus you need no special handling for the non error return case.

See http://support.microsoft.com/default.aspx?scid=kb;en-us;Q192800 for details.

Answer 2

First of all break the call into more clear logic:

int nRet = WSASend(m_socket, socketData->SetBuffer(socketData->GenerateLoginRequestHeader()), 1, NULL, NULL, reinterpret_cast<OVERLAPPED*>(socketData), NULL);
if (nRet == SOCKET_ERROR)
{
    if ((WSAGetLastError()) == WSA_IO_PENDING)
        nRet = 0; // ok
    else
        throw std::exception("Failed to send data."); // failed
}

Also, as you can see in my code, you should NOT pass the "&bytesSent" parameter according to WSASend:

Use NULL for this parameter if the lpOverlapped parameter is not NULL to avoid potentially erroneous results.

Besides that your call to WSASend() looks fine.