Can we do a smart-copy in Python?

Question

I'm a python newbie, and I'm writing a script to copy compiled files from one location to another. What I have is quite simple at the moment, something like this:

import os
import shutil

shutil.copy2 (src, dst)
#... many more shutil.copy commands
#src is a filename string
#dst is the directory where the file is to be copied

My problem is that many of the files being copied are large files, and not all of them are re-compiled in every compile cycle. Ideally, I would like to copy only the changed files in this script. Is there any way I can do this?

Answer 1

you could give this python implementation of rsync a try

http://freshmeat.net/projects/pysync/

Answer 2

You could make use of the file modification time, if that's enough for you:

# If more than 1 second difference
if os.stat(dest).st_mtime - os.stat(src).st_mtime > 1:
    shutil.copy2 (src, dst)

Or call a synchronization tool like rsync.

Answer 3

If you don't have a definite reason for needing to code this yourself in python, I'd suggest using rsync. From it's man-page:

Rsync is a fast and extraordinarily versatile file copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.

If you do wish to code this in Python, however, then the place to begin would be to study filecmp.cmp

Answer 4

How would you like to look for changed files? You can just use os.path.getmtime(path) on the src and check whether that's newer than some stored timestamp (the last time you copied for instance) or use a filecmp.cmp(f1, f2[, shallow]) to check whether a file is newer.

Watch out with filecmp.cmp, you also copy the stat (copy2) so you have to check wether a shallow compare is good enough for you.