Problem with libcurl cookie engine

Question

[Cross-posted from lib-curl mailing list]

I have a single threaded app (MSVC C++ 2005) build against a static LIBCURL 7.19.4

A test application connects to an in house server & performs a bespoke authentication process that includes posting a couple of forms, and when this succeeds creates a new resource (POST) and then updates the resource (PUT) using If-Match.

I only use a single connection to libcurl (i.e. only one CURL*)

The cookie engine is enabled from the start using curl_easy_setopt(CURLOPT_COOKIEFILE, "")

The cookie cache is cleared at the end of the authentication process using curl_easy_setopt(CURLOPT_COOKIELIST, "SESS"). This is required by the authentication process.

The next call, which completes a successful authentication, results in a couple of security cookies being returned from the server - they have no expiry date set.

The server (and I) expect the security cookies to then be sent with all subsequent requests to the server. The problem is that sometimes they are sent and sometimes they aren't.

I'm not a CURL expert, so I'm probably doing something wrong, but I can't figure out what. Running the test app in a loop results shows a random distribution of correct cookie handling.

As a workaround I've disabled the cookie engine and am doing basic manual cookie handling. Like this it works as expected, but I'd prefer to use the library if possible.

Does anyone have any ideas?

Thanks Seb

Answer 1

We've experienced issues with libcurl losing "session" when the headers are a particular size.

The two known cases we've seen are 1425 and 2885.

When the sent headers are this specific size the server doesn't appear to receive the proper cookies. We haven't actually tested against a controlled server to see what is actually received by the server.

The work-around we came up with was to alter the User-Agent slightly by adding a space at the end to change the header size.

Here is some code to predict the header size before the request is sent

size_t PredictHeaderOutSize(CURL *curl, bool doPost, const char* request, char* userAgent, const char* host, const char* form)
{
    size_t predictedHeaderOutSize = 0;

    // Note, so far predicting 1 byte per newline, fix the hard coded #'s below if that turns out to be wrong

    // POST/GET line
    predictedHeaderOutSize += (doPost ? 4 : 3); // POST vs GET
    predictedHeaderOutSize += strlen(request);
    predictedHeaderOutSize += 11; // Extra characters in 'POST <request> HTTP/1.1' not accounted for above

    // User-Agent line
    predictedHeaderOutSize += strlen(userAgent);
    predictedHeaderOutSize += 13;

    // Host: header
    predictedHeaderOutSize += strlen(host);
    predictedHeaderOutSize += 7;

    // Accept: */*
    predictedHeaderOutSize += 12;

    // Cookie:
    struct curl_slist *cookies=NULL;
    struct curl_slist *next_cookie;
    int num_cookies = 0;
    CURLcode res = curl_easy_getinfo(curl, CURLINFO_COOKIELIST, &cookies);
    if (res == CURLE_OK)
    {
        if (cookies != NULL)
        {
            // At least 1 cookie so add the extra space taken on cookie line
            predictedHeaderOutSize += 7;
            next_cookie = cookies;
            num_cookies = 1;
            while (next_cookie)
            {
                std::vector<std::string> cookie = QueueHelper::Split("\t", next_cookie->data, 7);
                if (cookie.size() != 7)
                {
                    // wtf?
                }
                else
                {
                    // For each cookie we add length of key + value + 3 (for the = ; and extra space)
                    predictedHeaderOutSize += cookie[5].length() + cookie[6].length() + 3;
                }
                next_cookie = next_cookie->next;
                num_cookies++;
            }
            curl_slist_free_all(cookies);
        }
    }
    else
    {
        printf("curl_easy_getinfo failed: %s\n", curl_easy_strerror(res));
    }

    if (doPost)
    {
        // Content-Length:
        size_t formLength = strlen(form);
        if (formLength < 10)
            predictedHeaderOutSize += 1;
        if (formLength >= 10 && formLength < 100)
            predictedHeaderOutSize += 2;
        if (formLength >= 100 && formLength < 1000)
            predictedHeaderOutSize += 3;
        if (formLength >= 1000 && formLength < 10000)
            predictedHeaderOutSize += 4;
        predictedHeaderOutSize += 17;

        // Content-Type: application/x-www-form-urlencoded
        predictedHeaderOutSize += 48;
    }

    predictedHeaderOutSize += 2; // 2 newlines at the end? something else? not sure

    return predictedHeaderOutSize;
}