Download only new/modified files with perl (or wget)

Question

I have a Perl script which downloads a large number of files from a remote server. I'd like to avoid hammering the server, so I'd like to avoid downloading a file if it hasn't been modified since my last check. Is there a good way to do this, either in Perl or with a shell script?

Can I get the server to send HTTP 304 rather than HTTP 200 for unmodified files?

Answer 1

Yes, use LWP::UserAgent and pay special attention to the mirror method. This is also available in the procedural LWP::Simple as the mirror function.

From LWP's POD:

This method will get the document identified by $url and store it in file called $filename. If the file already exists, then the request will contain an "If-Modified-Since" header matching the modification time of the file. If the document on the server has not changed since this time, then nothing happens. If the document has been updated, it will be downloaded again. The modification time of the file will be forced to match that of the server.

The return value is the the response object.

HTTP 304 is the response code the server will return if you pass the If-Modified-Since test and your copy is fresh. LWP does this internally with mirror -- you needn't worry about it.

Answer 2

This is based on Evan Carrol's answer, but I'm going to elaborate in case this is useful for someone else. I stubbed out the response section; I doubt that part of my code will be interesting.

#!/usr/bin/perl -w

require HTTP::Date;
require LWP::UserAgent;
require Date::Parse;

my $lastChecked = '2009-01-01';
my $ua = LWP::UserAgent->new;
$ua->default_header('If-Modified-Since' => HTTP::Date::time2str(Date::Parse::str2time($lastChecked)));

my $response = $ua->get('http://example.com/');

if ($response->code == 304) {
    print "No changes.\n";
} elsif ($response->is_success) {
    print $response->decoded_content;
} else {
    print "Response was error " . $response->code . ": '" . $response->status_line . "'\n";
}