Uploading large video via Youtube API causing Out of Memory

Question

I'm using PHP to send videos via Direct Upload to Youtube. It works fine for smaller sized videos but when trying to send a 390 MB video, I get the following error:

PHP Fatal error: Out of memory (allocated 3932160) (tried to allocate 390201902 bytes)

I've tried increasing memory_limit but that does not help.

    if ($isFile) {
        ini_set('memory_limit', '2G')
        $data = file_get_contents($data);
    }

    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_POST, 1);
    curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $data);

    $out = curl_exec($ch);
    curl_close($ch);

    return $out;

I've also tried running curl through exec() but then even weirder things happen:

curl http://uploads.gdata.youtube.com/feeds/api/users/default/uploads -H 'POST /feeds/api/users/default/uploads HTTP/1.1' -H 'Host: uploads.gdata.youtube.com' -H 'Authorization: OAuth [snip oauth info]"' -H 'GData-Version: 2' -H 'X-GData-Client: www.mywebsite.com' -H 'X-GData-Key: key=[snip]' -H 'Slug: video.AVI' -H 'Content-Type: multipart/related; boundary="iUI5C0hzisAHkx9SvaRJ"' -H 'Content-Length: 390193710' -H 'Connection: close' -d /tmp/youtube.xml

/tmp/youtube.xml is where I have saved the data file to upload. Perhaps this usage is wrong?

This will take about 6 minutes so it looks like the file is being sent but then I get an empty reply:

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed

 0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
 0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
 ...
 0     0    0     0    0     0      0      0 --:--:--  0:06:00 --:--:--     0
curl: (52) Empty reply from server

EDIT:

I am using OAuth so I cannot use the normal PHP API library for this. I must upload an XML file with the video's binary data included in the XML file as outlined here

I found another person with the same problem who had supplied code to read and send the file in chunks. However, when trying this method, Youtube would return a 411 page saying the "Content-Length" header is required. I am setting the content-length header so this could be a bug. This method uses the fsockopen() function instead of cURL. [Actually, looking at the code again, I realise I was just separating the headers with "\n" instead of "\r\n." That might be the issue. I will try with the carriage returns as well]

Edit 2:

I think the "\r\n" worked, but now with the code, I am again receiving an empty reply from Youtube.

Any Curl experts out there that can help me get this working? I'm completely stumped by this.

Answer 1

Try doing:

ini_set('memory_limit', -1);

Does it work?

Answer 2

How about

$filename = "--REMOTE FILE--";
$localfile = "/storage/local.flv";

$handle = fopen($filename, "r");

while ($contents = fread($handle, 10485760)) { // thats 10 MB

$localhandle = fopen($localfile, "a");
fwrite ($localhandle, $contents);
fclose($localhandle);

}

fclose($handle);

Answer 3

Try not to read the whole file into memory before sending. curl IMHO supports reading the file itself before uploading and therefor should work inside the memory borders. See the following blog post for an example: http://dtbaker.com.au/random-bits/uploading-a-file-using-curl-in-php.html

I haven't worked with the youtube direct upload API yet but after a quick look I saw that this does not seem the be a normal html form file upload but a bit more complex data format. I am not sure whether you can do this with plain cURL without building the whole POST data in memory yourself.

If you have this small memory limits (~4 MB of RAM) you could try building your own simple HTTP client on top of the streams API in PHP: Create a temporary file and write your POST request data into that file handle (using fwrite() for normal strings and stream_copy_to_stream() for direct file to file data transfer). When your request is ready, rewind your temporary file to the beginning and then copy that stream into the connection with the youtube http server (again using stream_copy_to_stream()).

As the streams are copied in small chunks you should be able to do that with less then 4 MB of RAM, even for large files.

EDIT:

Following pseudo-php-code-mashup should be helpful ;)

$xmlAPIRequest = /* fill with the XML-API-Request */
$boundaryString = /* fill me with some random data */

// create temporary file handle
$pdh = tmpfile();
fwrite($pdh, "--$boundaryString\r\n");
fwrite($pdh, "Content-Type: application/atom+xml; charset=UTF-8\r\n\r\n");
fwrite($pdh, $xmlAPIRequest."\r\n");
fwrite($pdh, "--$boundaryString\r\n");
fwrite($pdh, "Content-Type: <video_content_type>\r\nContent-Transfer-Encoding: binary\r\n\r\n");

$videoFile = fopen("/path/to/video", "r");
stream_copy_to_stream($videoFile, $pdh);
fclose($videoFile);

fwrite($pdh, "--$boundaryString--\r\n");
/* not quite sure, whether there needs to be another linebreak before the boundary string */

$info = fstat($pdh);
rewind($pdh);

$contentLength = $info['size'];

$conn = fsockopen("hostname", 80);
/* write http request to $conn and use $contentLength for Content-Length header */
/* after last header you put another line break to tell them that now the body follows */

// write post data from stream to stream
stream_copy_to_stream($pdh, $conn);

// ... process response... etc...

There are certainly a lot of bugs in this code but as it is only a short example I think we can live with that. ;)