Stream FTP download to output

Question

I am trying to stream/pipe a file to the user's browser through HTTP from FTP. That is, I am trying to print the contents of a file on an FTP server.

This is what I have so far:

public function echo_contents() {                    
    $file = fopen('php://output', 'w+');             

    if(!$file) {                                     
        throw new Exception('Unable to open output');
    }                                                

    try {                                            
        $this->ftp->get($this->path, $file);         
    } catch(Exception $e) {                          
        fclose($file);  // wtb finally               

        throw $e;                                    
    }                                                

    fclose($file);                                   
}

$this->ftp->get looks like this:

public function get($path, $stream) {
    ftp_fget($this->ftp, $stream, $path, FTP_BINARY);  // Line 200
}

With this approach, I am only able to send small files to the user's browser. For larger files, nothing gets printed and I get a fatal error (readable from Apache logs):

PHP Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to allocate 15994881 bytes) in /xxx/ftpconnection.php on line 200

I tried replacing php://output with php://stdout without success (nothing seems to be sent to the browser).

How can I efficiently download from FTP while sending that data to the browser at the same time?

Note: I would not like to use file_get_contents('ftp://user%3Apass@host%3Aport/path/to/file'); or similar.

Answer 1

Sounds like you need to turn off output buffering for that page, otherwise PHP will try to fit it in all memory.

An easy way to do this is something like:

while (ob_end_clean()) {
    ; # do nothing
}

Put that ahead of your call to ->get(), and I think that will resolve your issue.

Answer 2

Hi,

(I've never met this problem myself, so that's just a wild guess ; but, maybe... )

Maybe changing the size of the ouput buffer for the "file" you are writing to could help ?

For that, see stream_set_write_buffer.

For instance :

$file = fopen('php://output', 'w+');
stream_set_write_buffer($fp, 0);

With this, your code should use a non-buffered stream -- this might help...

Answer 3

Try:

@readfile('ftp://username:password@host/path/file'));

I find with a lot of file operations it's worthwhile letting the underlying OS functionality take care of it for you.

Answer 4

a quick search brought up php’s flush.

this article might also be of interest: http://www.net2ftp.org/forums/viewtopic.php?id=3774

Answer 5

Found a solution!

Create a socket pair (anonymous pipe?). Use the non-blocking ftp_nb_fget function to write to one end of the pipe, and echo the other end of the pipe.

Tested to be fast (easily 10MB/s on a 100Mbps connection) so there's not much I/O overhead.

Be sure to clear any output buffers. Frameworks commonly buffer your output.

public function echo_contents() {
    /* FTP writes to [0].  Data passed through from [1]. */
    $sockets = stream_socket_pair(STREAM_PF_UNIX, STREAM_SOCK_STREAM, STREAM_IPPROTO_IP);

    if($sockets === FALSE) {
        throw new Exception('Unable to create socket pair');
    }

    stream_set_write_buffer($sockets[0], 0);
    stream_set_timeout($sockets[1], 0);

    try {
        // $this->ftp is an FtpConnection
        $get = $this->ftp->get_non_blocking($this->path, $sockets[0]);

        while(!$get->is_finished()) {
            $contents = stream_get_contents($sockets[1]);

            if($contents !== false) {
                echo $contents;
                flush();
            }

            $get->resume();
        }

        $contents = stream_get_contents($sockets[1]);

        if($contents !== false) {
            echo $contents;
            flush();
        }
    } catch(Exception $e) {
        fclose($sockets[0]);    // wtb finally
        fclose($sockets[1]);

        throw $e;
    }

    fclose($sockets[0]);
    fclose($sockets[1]);
}

// class FtpConnection
public function get_non_blocking($path, $stream) {
    // $this->ftp is the FTP resource returned by ftp_connect
    return new FtpNonBlockingRequest($this->ftp, $path, $stream);
}

/* TODO Error handling. */
class FtpNonBlockingRequest {
    protected $ftp = NULL;
    protected $status = NULL;

    public function __construct($ftp, $path, $stream) {
        $this->ftp = $ftp;

        $this->status = ftp_nb_fget($this->ftp, $stream, $path, FTP_BINARY);
    }

    public function is_finished() {
        return $this->status !== FTP_MOREDATA;
    }

    public function resume() {
        if($this->is_finished()) {
            throw BadMethodCallException('Cannot continue download; already finished');
        }

        $this->status = ftp_nb_continue($this->ftp);
    }
}