PHP FTP Upload thousands of files

Question

Hi,

I've written a small FTP class which I used to move files from a local server to a remote server. It does this by checking an array of local files with an array of files on the remote server. If the file exists on the remote server, it won't bother uploading it.

The script works fine for small amounts of files, but I've noticed that the local server can have as many as 3000+ image files to transfer, this seems to cause the script to flop and only transfer a 100 or so.

How can I modify the script to handle potentially thousands of image transfer files?

Answer 1

What could happen is that you take too long to execute the script (this doesn't apply to commandline php) if that happens your script will be stopped by the web server. You can change php settings to fix that, but that will not scale very well (because your browser will eventually time out too). Perhaps running the script from commandline (called cli php) will work.

It sounds to me like you are implementing something that already exists. You should have a look at rsync (for linux) if you have control of both servers.

Answer 2

Run cron more frequently, and limit the script to an upload of 80 images per run.

Answer 3

If the problem is with php or browser time out you can create a file (example below) and cron it or call from a browser.

<?
echo "Running cli syncfiles.php";
system("&php syncfiles.php"); // & pushes file to background processing on linux 
?>

If you are having a problem because the ftp is throttling your connections, or is throttling your simultaneous uploads/downloads within x amount of time, then you can probably throw some kind of timers into the code.

<?
$counter=0;
for($i=0;$i<$numftpfiles;$i++)
{
   syncfile($i); // this represents your sync code
   usleep(250000); // sleep for 1/4 second
   $count++;
   if($count>50)
   {
     usleep(30000000); // sleep for 30 seconds
     $count=0;
   }
}
?>

Answer 4

You could zip then first in php http://www.php.net/manual/en/book.zip.php

Then upload one larger zip file. The total file size is unlikely to change, but I've found when doing transfers of a lot of files across my WAN it is faster anyway.

-Will