How to Ensure file copy integrity in Ubuntu

Question

In Ubuntu if a file /A/largefile.foo is moved via 'cp' or 'mv' to /B/largefile.foo, how can I ensure that the entire file is written before attempting to move the file a second time to /C/largefile.foo?

Case: 1. An remote automated Script X begins an rsync to destination folder A of a very large file. 2. A cron job on the local machine hourly moves files from folder A to folder B, but I do not want to copy partial files (I only want files that have finished uploading completely)

Case: 1. A cron job periodically moves files from folder B to folder C, but I do not want to copy partial files (I only want files that have finished uploading completely)

Note: I'm looking for a solution that does not require the addition of pid files

Answer 1

A common approach is to copy files with a ".part" suffix. When the copy is done, the script renames the file to remove the .part. Your other script simply has to ignore the .part file.

This is what download managers, FTP programs and such are doing.

Answer 2

For the first rsync case, you can use the --partial-dir option combined with --delay-updates, it will write partial files to a separate location, renaming them into place once done.

You should make sure that the partial directory is on the same filesystem as the destination, so the renames will be atomic.

for the second case use mv (again assuming one file system) which will be atomic.

see the rsync manual for more information

Answer 3

Move operation is atomic on POSIX within one file system.

If you open a file and then delete it from the file system, you can still write into the file. That is why we need to tell syslogd to reopen file handles after we rotate logs.

The solution is to write to a temporary file first and then move it to the destination. Option --delay-updates for rsync makes it do exactly this.

Answer 4

Use rsync. You can also md5sum the orig file and the new file on destination (if cp)