Whats the best way to transport a file to a different server?

Question

Im running a file host thats grown beyond the capacity of a single server, and I need to implement multiple server storage of files. I'd like to do this as cheap as possible, so those fancy mass storage methods are out of the question. I simply want to move a file thats uploaded by the user to the "gateway" server, which hosts all the http and mysql, to one of the media servers. It can either be done at the end of the user's request, or via cron every couple of minutes.

At this point the only method Im truly familiar with, is using ftp_put php function, and simply ftping the file to a different server, but I've had problems with this method in the past, especially for larger files, and a lot of the files that will be getting transferred will be over 100mb.

Can anyone suggest a good solution for this? Preferably Im looking for a purely software solution... hopefully nothing more than a php/bash script.

Answer 1

you can write a bash script that run in cron

and use command line like scp or sftp or rsync

example :

[bash]# scp filename [email protected]:/home/alice

Copy "filename" into alice's home directory on the remote galaxy.example.com server

Answer 2

One method that requires no programmatic setup, and is quite secure is to mount a folder from the other server, and use php to save to that directory like you would normally.

You don't have to use sshfs, there are a bunch of other ways to provide the same solution. I would use sshfs for this situation as it works across ssh, and therefore is secure and relatively easy to setup.

To achieve this (using sshfs):

1. Install sshfs
2. Mount a ssfs file to a folder accessable to php
3. Use a php script to store the files within the sshfs mount, and therefore on the other server.

Another method is to setup an rsync command between the two using crontab.

Answer 3

Use cron, a bash script, and rsync.

cron: "Cron is a time-based job scheduler in Unix-like computer operating systems. 'cron' is short for 'chronograph'."

bash: "Bash is a free software Unix shell written for the GNU Project."

rsync: "rsync is a software application for Unix systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate."

In summary, place the rsync command in a bash script and get cron to run it at regular intervals.

See here for some examples: