EPIPE blocks server

Question

I have written a single-threaded asynchronous server in C running on Linux: The socket is non-blocking and as for polling, I am using epoll. Benchmarks show that the server performs fine and according to Valgrind, there are no memory leaks or other problems.

The only problem is that when a write() command is interrupted (because the client closed the connection), the server will encounter an EPIPE. I am doing the interrupted artificially by running the benchmarking utility "siege" with the parameter -b. It does lots of requests in a row which all work perfectly. Now I press CTRL-C and restart the "siege". Sometimes I am lucky and the server does not manage to send the full response because the client's fd is invalid. As expected, errno is set to EPIPE. I handle this situation, execute close() on the fd and then free the memory related to the connection. Now the problem is that the server blocks and does not answer properly anymore. Here is the strace output:

epoll_wait(4, {{EPOLLIN, {u32=0, u64=0}}}, 128, -1) = 1
accept(3, {sa_family=AF_INET, sin_port=htons(55328), sin_addr=inet_addr("127.0.0.1")}, [16]) = 5
fcntl64(5, F_GETFL)                     = 0x2 (flags O_RDWR)
fcntl64(5, F_SETFL, O_RDWR|O_NONBLOCK)  = 0
epoll_ctl(4, EPOLL_CTL_ADD, 5, {EPOLLIN|EPOLLERR|EPOLLHUP|EPOLLET, {u32=144039912, u64=144039912}}) = 0
epoll_wait(4, {{EPOLLIN, {u32=144039912, u64=144039912}}}, 128, -1) = 1
read(5, "GET /user/register HTTP/1.1\r\nHos"..., 4096) = 161
send(5, "HTTP/1.1 200 OK\r\nContent-Type: t"..., 106, MSG_NOSIGNAL) = 106 <<<<
send(5, "00001000\r\n", 10, MSG_NOSIGNAL) = -1 EPIPE (Broken pipe)        <<<< Why did the previous send() work?
close(5)                                = 0
epoll_wait(4, {{EPOLLIN, {u32=0, u64=0}}}, 128, -1) = 1
accept(3, {sa_family=AF_INET, sin_port=htons(55329), sin_addr=inet_addr("127.0.0.1")}, [16]) = 5
...

(I deleted the printf()'s from the log in case you're wondering)

As you can see, the client establishes a new connection which consequently is accepted. Then, it's added to the EPOLL queue. epoll_wait() signalises that the client sent data (EPOLLIN). The request is parsed and and a response is composed. Sending the headers works fine but when it comes to the body, write() results in an EPIPE. It is not a bug in "siege" because it blocks any incoming connections, no matter from which client.

#include "Connection.h"

static ExceptionManager *exc;

void Connection0(ExceptionManager *e) {
    exc = e;
}

void Connection_Init(Connection *this) {
    Socket_Init(&this->server);
    Socket_SetReusableFlag(&this->server);
    Socket_SetCloexecFlag(&this->server, true);
    Socket_SetBlockingFlag(&this->server, true);
    Socket_ListenTCP(&this->server, 8080, SOMAXCONN);

    // Add the server socket to epoll
    Poll_Init(&this->poll, SOMAXCONN, (void *) &Connection_OnEvent, this);
    Poll_AddEvent(&this->poll, NULL, this->server.fd, EPOLLIN | EPOLLET | EPOLLHUP | EPOLLERR);

    this->activeConn = 0;
}

void Connection_Destroy(Connection *this) {
    Poll_Destroy(&this->poll);
    Socket_Destroy(&this->server);
}

void Connection_Process(Connection *this) {
    Poll_Process(&this->poll, -1);
}

void Connection_AcceptClient(Connection *this) {
    Client *client;

    SocketConnection conn = Socket_Accept(&this->server);
    SocketConnection_SetBlockingFlag(&conn, true);

    client = New(Client);

    client->req = NULL;

    client->conn = New(SocketConnection);
    client->conn->remote = conn.remote;
    client->conn->fd = conn.fd;

    this->activeConn++;

    Poll_AddEvent(&this->poll, client, conn.fd, EPOLLIN | EPOLLET | EPOLLHUP | EPOLLERR);
}

void Connection_DestroyClient(Connection *this, Client *client) {
    // Poll_DeleteEvent(&this->poll, client->conn->fd);
    SocketConnection_Close(client->conn);

    if (client->req != NULL) {
        Request_Destroy(client->req);
        Memory_Free(client->req);
    }

    if (client->conn != NULL) {
        Memory_Free(client->conn);
    }

    Memory_Free(client);

    this->activeConn--;
}

void Connection_OnEvent(Connection *this, int events, Client *client) {
    /* error or connection hung up */
    if (client != NULL && (events & (EPOLLHUP | EPOLLERR))) {
        String_Print(String("EPOLLHUP | EPOLLERR received\n"));
        Connection_DestroyClient(this, client);
        return;
    }

    /* incoming connection */
    if (client == NULL && (events & EPOLLIN)) {
        if (this->activeConn > SOMAXCONN - 1) { /* TODO */
            String_Print(String("Too many connections...\n"));
            return;
        }

        Connection_AcceptClient(this);
        return;
    }

    /* receiving data from client */
    if (client != NULL && (events & EPOLLIN)) {
        if (client->req == NULL) {
            client->req = New(Request);
            Request_Init(client->req, client->conn);
        }

        bool keepOpen = false;

        try (exc) {
            keepOpen = Request_Parse(client->req);
        } catch(&SocketConnection_PipeException, e) {
            printf("Caught PipeException on fd=%d\n", client->conn->fd); fflush(stdout);
        } catch(&SocketConnection_ConnectionResetException, e) {
            printf("Caught ConnectionResetException on fd=%d\n", client->conn->fd); fflush(stdout);
        } finally {
            if (!keepOpen) {
                printf("Will close...\n"); fflush(stdout);
                Connection_DestroyClient(this, client);
            }
        } tryEnd;
    }
}

Answer 1

Use sigaction() to set the action for SIGPIPE to SIG_IGN. Then you will just get the return code of -1 with errno set to EPIPE, without the signal.

On Linux, an alternative is to use send() with the MSG_NOSIGNAL flag, instead of write(). This allows you to suppress the signal for that write, without affecting any others. An alternative on BSD systems is the SO_NOSIGPIPE socket option, which suppresses SIGPIPE for all writes on that socket.

At any time the kernel TCP implementation may receive a TCP RST from the peer. The next write on the socket after that point will result in the EPIPE error, and the SIGPIPE signal if it was not suppressed. So even with two consecutive writes, the first could be successful and the next one could fail with EPIPE and SIGPIPE.

Update: Although it is not part of the code that was posted, I see in your strace output that you are calling fcntl64(5, F_SETFL, O_RDONLY|O_NONBLOCK). Since O_RDONLY is 0, probably your code just sets the flags to O_NONBLOCK. It should get the current flags and then set it to OldFlags | O_NONBLOCK in order to set non-blocking mode. Setting a socket to read-only mode seems likely to cause a problem when writing. Or it may be that you accidentally used F_GETFD instead of F_GETFL to get the old flags.

Answer 2

I can't tell from your code (possibly because I've not looked in the right place) whether you adjust the epoll structure after you get the SIGPIPE from one of the descriptors. After you get a SIGPIPE (or EPIPE) from a file descriptor, you will get a repeat on any subsequent use of the same file descriptor. You'd need to close the file descriptor, and then adjust your server's internal structures appropriately.