Is it practical in an application to download data from an external web resource when local data isn't available?

Question

Awhile ago I came across a script that basically fetched a list of countries/states from a web resource if it wasn't located in a database, and this script would then populate the database with those contents and from then on, rely on them from then on.

Since I'm working on a localization class of my own, I'll be using the same locale data Zend is using, in the form of around ~60 or so xml files which contain localised data such as countries, languages for locales.

I figure since the framework I'm working on will rely on these files from now on ( where it isn't now ), and none of the servers now have this data, should I:

• Setup my web application to download these files from a central server where all the content is stored in a .tar.gz, unpack them, store them on the server and then rely on them
• Create a separate script to do this, and not actually do this within the application.

Pseudo code:

if ( !data ) {
    resource = getFile( 'http://central-server.com/tar.gz' );
    if ( resource ) {
        resource = unpack( directory, resource )
        return true
    }
    throw Exception('could not download files.')
}

Answer 1

if this is a library, i would probably have this be part of the setup steps. an error can be printed if data isn't there.

Answer 2

Have an install script do the downloading, or throw an error if its not available. Downloading as requested from the server could lead to timeouts and would likely turn away users. fsockopen is the easiest way to do this and deal with sockets by hand if you don't have CURL setup and can't fopen/fread remote files.

Answer 3

I would go for the first option iff the data needs to be contantly updated, otherwise I would choose your second option.

---

Here is a method I developed some years ago, that was part of a GeoIP class:

function Update()
{
    $result = false;

    $databases = glob(HIVE_DIR . 'application/repository/GeoIP/GeoIP_*.dat');

    foreach ($databases as $key => $value)
    {
     $databases[$key] = basename($value);
    }

    $databases[] = 'GeoIP.dat.gz';

    $date = date('ym');

    if ((!in_array('GeoIP_' . $date . '.dat', $databases)) && (date('j') >= 2))
    {
     if ($this->Hive->Filesystem->Write(HIVE_DIR . 'application/repository/GeoIP/GeoIP.dat.gz', file_get_contents('http://www.maxmind.com/download/geoip/database/GeoIP.dat.gz'), false) === true)
     {
      $handler = gzopen(HIVE_DIR . 'application/repository/GeoIP/GeoIP.dat.gz', 'rb');

      $result = $this->Hive->Filesystem->Write(HIVE_DIR . 'application/repository/GeoIP/GeoIP_' . $date . '.dat', gzread($handler, 2 * 1024 * 1024), false);

      gzclose($handler);

      foreach ($databases as $database)
      {
       $this->Hive->Filesystem->Delete(HIVE_DIR . 'application/repository/GeoIP/' . $database);
      }
     }
    }

    return $result;
}

Basically the Update() was executed every single time, it would then check if the day of the month equal or higher than 2 (MaxMind releases GeoIP databases on the first day of the month) and if a a database for that month didn't existed already. Only if both these conditions where true the method would download, unpack, rename the database and remove all the old databases from previous months.

In your case, since you're dealing with locales, doing a periodical check similar to this once in a while might not be a bad idea, since countries change stuff (names, currencies, calling codes, etc...) a lot.