Transfer files using SFTP and either PHP or shell/terminal script

Question

I need to write a script that is run as a cron job every night which transfers some report files via sftp to another server.

The report files are created every night using another cron in the format 'support_[date].csv' & 'download_[date].csv'.

I'm wondering if you had any pointers on how to do the following:

1. Find the 2 files created on latest [date]
2. Copy these files to another server using SFTP

I've tried several PHP scripts utilising the ssh2 extension, but to no avail. Is there a way to do it using a shell script? It's not something I am hugely familiar with to be honest (hence going down the PHP route initially)

This was one of my PHP scripts which didn't work:

$src = 'test.csv';

$filename = 'test.csv';
$dest = '/destination_directory_on_server/'.$filename;

$connection = ssh2_connect('example.com', 22);
ssh2_auth_password($connection, 'username', 'password');

// Create SFTP session
$sftp = ssh2_sftp($connection);

$sftpStream = fopen('ssh2.sftp://'.$sftp.$dest, 'w');

try {

            if (!$sftpStream) {
                throw new Exception("Could not open remote file: $dest<br>");
            }

            $data_to_send = file_get_contents($src);

            if ($data_to_send === false) {
                throw new Exception("Could not open local file: $src.<br>");
            }

            if (fwrite($sftpStream, $data_to_send) === false) {
                throw new Exception("Could not send data from file: $src.<br>");
            } else {
                //Upload was successful, post-upload actions go here...
            }

            fclose($sftpStream);

        } catch (Exception $e) {

            //error_log('Exception: ' . $e->getMessage());
           echo 'Exception: ' . $e->getMessage();
            if($sftpStream) {fclose($sftpStream);}

        }

This were the error messages I got:

Warning: fopen() [function.fopen]: URL file-access is disabled in the server configuration in /path_to_script/sftp-test.php on line 17

Warning: fopen(ssh2.sftp://Resource id

3/destination_directory_on_server/test.csv)

[function.fopen]: failed to open stream: no suitable wrapper could be found in /path_to_script/sftp-test.php on line 17 Exception: Could not open remote file: /destination_directory_on_server/test.csv

Answer 1

The answer is right there, in the error message:

Warning: fopen() [function.fopen]: URL file-access is disabled in the server configuration

means that file-access through URL wrappers is disabled in the server configuration.

Check your PHP config, especially allow_url_fopen. PHP documentation says "This setting can only be set in php.ini due to security reasons", so check it there.

See also fopen: "If PHP has decided that filename specifies a registered protocol, and that protocol is registered as a network URL, PHP will check to make sure that allow_url_fopen is enabled. If it is switched off, PHP will emit a warning and the fopen call will fail." As far as I can tell, that's exactly what is happening there.

If you can't or won't enable allow_url_fopen, you still have some options:

• call sftp directly
• mount a share with sshfs and then use it as a normal folder

Answer 2

This will not work from PHP from your server because your php.ini has disabled remote wrappers

allow_url_fopen boolean

This option enables the URL-aware fopen wrappers that enable accessing URL object like files. Default wrappers are provided for the access of remote files using the ftp or http protocol, some extensions like zlib may register additional wrappers.

Note: This setting can only be set in php.ini due to security reasons.

However, you could simply let your cron job call a shell script that that uses sftp or rsync directly. You don't have to do this with PHP.

I'm voting to move this to ServerFault to get better support for shell scripting.

Answer 3

using the terminal to find latest date of your file, you can use ls -1tr . Then use scp (not sftp) to copy/transfer files over example,

#!/bin/bash
latest_download=$(ls -1tr download*csv | tail -1)
latest_support=$(ls -1tr support*csv | tail -1)
scp $latest_download [email protected]:somedir  # syntax from memory, check man page for correct syntax
scp $latest_support [email protected]:somedir

check the man page of scp for usage

Answer 4

Muchos kudos to ghostdog74! Managed to get this working, but with sftp.

First I managed to set up key authentication, then partly using ghostdog74's script I did this and it worked perfectly!

cd /directorywithfilesin
latest_download=$(ls -1tr download* | tail -1)
latest_support=$(ls -1tr support* | tail -1)
sftp [email protected] <<EOF
cd /dir_to_copy_to
put $latest_download
put $latest_support
EOF

Thanks!