Make backup of large site with 100,000+ files/images

Question

I tried backing up our site today using the Unix 'cp' command and ended up getting our office blocked out by PLESK -> it added my ip to /etc/hosts.deny as it thought I was flooding the server.

After Tech support fixed the issue, they suggested I go folder by folder to back it up, but there's about 10,000 folders on the site totaling 1/2 a terabyte, each with multiple sub-folders, so this isn't viable.

Basically I want to be able to mirror the domain on another domain we've got set up on the same dedicated server so I can test with live images (the bulk of our content).

Any suggestions e.g adding some rules to open_base_dir and getting PHP to recursively copy the folders to the other domain (remember it's on the same dedicated box so it just needs to traverse the directory, not FTP things).

Answer 1

You could try rsync with the --bwlimit option. Or you could do a one-off sneakernet backup with a USB drive and subsequently maintain the backup with rsync.

As a general rule, always use rsync in preference to cp. It has many benefits, including the fact that you don't have to mount remote volumes.