PHP HTTP POST fails when cURL data > 1024

Question

Note: solution at the end

If I attempt to do a HTTP POST of over 1024 characters, it fails. Why? Here is a minimal example:

recipient.php:

<?php
if (strlen(file_get_contents('php://input')) > 1000
    || strlen($HTTP_RAW_POST_DATA) > 1000) {
 echo "This was a triumph.";
}
?>

sender.php:

<?php
function try_to_post($char_count) {
 $url = 'http://gpx3quaa.joyent.us/test/recipient.php';
 $post_data = str_repeat('x', $char_count);
 $c = curl_init();
 curl_setopt_array($c,
                    array(  CURLOPT_URL => $url,
                            CURLOPT_HEADER => false,
                            CURLOPT_CONNECTTIMEOUT => 999,
                            CURLOPT_RETURNTRANSFER => true,
                            CURLOPT_POST => 1,
                            CURLOPT_POSTFIELDS => $post_data
                    )
 );
 $result = curl_exec($c);
 echo "{$result}\n";
 curl_close($c);
}

for ($i=1020;$i<1030;$i++) {
 echo "Trying {$i} - ";
 try_to_post($i);
}
?>

output:

Trying 1020 - This was a triumph.
Trying 1021 - This was a triumph.
Trying 1022 - This was a triumph.
Trying 1023 - This was a triumph.
Trying 1024 - This was a triumph.
Trying 1025 - 
Trying 1026 - 
Trying 1027 - 
Trying 1028 - 
Trying 1029 -

configuration:

PHP Version 5.2.6
libcurl/7.18.0 OpenSSL/0.9.8g zlib/1.2.3 libidn/1.8
lighttpd-1.4.19

Solution

Add the following option for cURL:

curl_setopt($ch,CURLOPT_HTTPHEADER,array("Expect:"));

The reason seems to be that any POST over 1024 character causes the "Expect: 100-continue" HTTP header to be sent, and Lighttpd 1.4.* does not support it. I found a ticket for it: http://redmine.lighttpd.net/issues/show/1017

They say it works in 1.5.

Answer 1

First thoughts...

The manual page for curl_setopt says of CURLOPT_POSTFIELDS

"The full data to post in a HTTP "POST" operation. To post a file, prepend a filename with @ and use the full path. This can either be passed as a urlencoded string like 'para1=val1&para2=val2&...' or as an array with the field name as key and field data as value."

Could it be that your value is being treated as if it were urlencoded, and thus looks like a big long name with no value. Something somewhere is deciding to truncate that name.

Maybe you could alter it to something like

$post_data = "data=".str_repeat('x', $char_count);

Turns out this was too easy, and the problem was a little deeper. So, how to debug?

Find out exactly what CURL sends to the server

Another debugging tactic might be to formulate a curl command line which achieves the same thing, and have it output the HTTP request details as it makes them.

Testing the server by hand

You can eliminate the server from the equation by perform a request by hand, e.g. telnetting to port 80 on your server and sending it a request >1024 chars

POST /test/recipient.php HTTP/1.0
Host: gpx3quaa.joyent.us
Content-Length:1028

xxxxx(I put 1028 chars here, no point copying them all here!)

I got this response

HTTP/1.0 200 OK
Connection: close
Content-type: text/html; charset=UTF-8
Content-Length: 19
Date: Tue, 20 Jan 2009 21:35:16 GMT
Server: lighttpd/1.4.19

This was a triumph.Connection closed by foreign host.

So at least you now know it's all on the client side, possible some CURL option or configuration setting somewhere :(

Final Answer!

The problem intrigued me so I dug deeper

If you use CURLOPT_VERBOSE=>true, you'll see that CURL sends an extra header on the bigger posts:Expect: 100-Continue. Your lighttpd server doesn't like this, it would seem.

You can stop CURL from doing this by forcing it to use HTTP/1.0 with CURLOPT_HTTP_VERSION=>CURL_HTTP_VERSION_1_0 in your curl_setopt options array.

Answer 2

You can convince PHP's curl backend to stop doing the 100-continue-thing by setting an explicit request header:

curl_setopt($ch, CURLOPT_HTTPHEADER, array('Expect:'));

This way you can post a request however long you would ever want and curl will not do the dual phase post.

I've blogged about this nearly two years ago.

Answer 3

I had a similar problem with a IIS server, using SSL v3.

I kept getting the following cURL error when CURLOPT_POSTFIELDS was longer than 1024 :

52 - SSL read: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number, errno 0

Adding CURLOPT_HTTPHEADER : "Expect:" solved the problem for me.

Thank you so much for this thread!