How can I handle template dependencies in Template Toolkit?

Question

My static web pages are built from a huge bunch of templates which are inter-included using Template Toolkit's "import" and "include", so page.html looks like this:

[% INCLUDE top %]
[% IMPORT middle %]

Then top might have even more files included.

I have very many of these files, and they have to be run through to create the web pages in various languages (English, French, etc., not computer languages). This is a very complicated process and when one file is updated I would like to be able to automatically remake only the necessary files, using a makefile or something similar.

Are there any tools like makedepend for C files which can parse template toolkit templates and create a dependency list for use in a makefile?

Or are there better ways to automate this process?

Answer 1

Template Toolkit does come with its own command line script called ttree for building TT websites ala make.

Here is an ttree.cfg file I use often use on TT website projects here on my Mac:

# directories
src = ./src
lib = ./lib
lib = ./content
dest = ./html

# pre process these site file
pre_process = site.tt

# copy these files
copy = \.(png|gif|jpg)$

# ignore following
ignore = \b(CVS|RCS)\b
ignore = ^#
ignore = ^\.DS_Store$
ignore = ^._

# other options
verbose
recurse

Just running ttree -f ttree.cfg will rebuild the site in dest only updating whats been changed at source (in src) or in my libraries (in lib).

For more fine grained dependencies have a look a Template Dependencies.

Update - And here is my stab at getting dependency list by subclassing Template::Provider:

{
    package MyProvider;
    use base 'Template::Provider';

    # see _dump_cache in Template::Provider
    sub _dump_deps {
        my $self = shift;

        if (my $node = $self->{ HEAD }) {
            while ($node) {
                my ($prev, $name, $data, $load, $next) = @$node;

                say {*STDERR} "$name called from " . $data->{caller}
                    if exists $data->{caller};

                $node = $node->[ 4 ];
            }
        }
    }
}


use Template;

my $provider = MyProvider->new;

my $tt = Template->new({
    LOAD_TEMPLATES => $provider,
});

$tt->process( 'root.tt', {} ) or die $tt->error;

$provider->_dump_deps;

The code above displays all dependencies called (via INCLUDE, INSERT, PROCESS and WRAPPER) and where called from within the whole root.tt tree. So from this you could build a ttree dependency file.

/I3az/

Answer 2

In case all you care about are finding file names mentioned in directives such as INCLUDE, PROCESS, WRAPPER etc, one imagine even using sed or perl from the command line to generate the dependencies.

However, if there are subtler dependencies (e.g., you reference an image using <img> in your HTML document whose size is calculated using the Image plugin, the problem can become much less tractable.

I haven't really tested it but something like the following might work:

#!/usr/bin/perl

use strict; use warnings;

use File::Find;
use File::Slurp;
use Regex::PreSuf;

my ($top) = @ARGV;

my $directive_re = presuf qw( INCLUDE IMPORT PROCESS );

my $re = qr{
    \[%-? \s+ $directive_re \s+ (\S.+) \s+ -?%\]
}x;

find(\&wanted => $top);

sub wanted {
    return unless /\.html\z/i;

    my $doc = read_file $File::Find::name;
    printf "%s : %s\n", $_, join(" \\\n", $doc =~ /$re/g );
}

Answer 3

After reading the ttree documentation, I decided to create something myself. I'm posting it here in case it's useful to the next person who comes along. However, this is not a general solution, but one which works only for a few limited cases. It worked for this project since all the files are in the same directory and there are no duplicate includes. I've documented the deficiencies as comments before each of the routines.

If there is a simple way to do what this does with ttree which I missed, please let me know.

my @dependencies = make_depend ("first_file.html.tmpl");

# Bugs:
# Insists files end with .tmpl (mine all do)
# Does not check the final list for duplicates.

sub make_depend
{
    my ($start_file) = @_;
    die unless $start_file && $start_file =~ /\.tmpl/ && -f $start_file;
    my $dir = $start_file;
    $dir =~ s:/[^/]*$::;
    $start_file =~ s:\Q$dir/::;
    my @found_files;
    find_files ([$start_file], \@found_files, $dir);
    return @found_files;
}

# Bugs:
# Doesn't check for including the same file twice.
# Doesn't allow for a list of directories or subdirectories to find the files.
# Warning about files which aren't found switched off, due to
# [% INCLUDE $file %]

sub find_files
{
    my ($files_ref, $foundfiles_ref, $dir) = @_;
    for my $file (@$files_ref) {
        my $full_name = "$dir/$file";
        if (-f $full_name) {
            push @$foundfiles_ref, $full_name;
            my @includes = get_includes ($full_name);
            if (@includes) {
                find_files (\@includes, $foundfiles_ref, $dir);
            }
        } else {
#            warn "$full_name not found";
        }
    }
}

# Only recognizes two includes, [% INCLUDE abc.tmpl %] and [% INCLUDE "abc.tmpl" %]

sub get_includes
{
    my ($start_file) = @_;
    my @includes;
    open my $input, "<", $start_file or die "Can't open $start_file: $!";
    while (<$input>) {
        while (/\[\%-?\s+INCLUDE\s+(?:"([^"]+)"|(.*))\s+-?\%\]/g) {
            my $filename = $1 ? $1 : $2;
            push @includes, $filename;
        }
    }
    close $input or die $!;
    return @includes;
}