How can I manage Perl module dependencies?

Question

I'm currently in a project which develops using a framework developed by another department as the base. We are currently introducing quality standards (at last, yay!) in our department, but it's currently impossible to introduce those to the other department. As a consequence, we are working against a constant moving target without either API stability or stable releases, which is stressful at the very least.

Since we are trying to fix things at our end first, we'd like to secure ourselves as far as it gets against changes in the "upstream" a.k.a. framework code. We'd envisioned hard module dependencies:

1. Using only certain version ranges of framework modules, defined in code.
2. Using a unit-test check to ensure that all necessary versions are still available.
3. Every version range extension requiring peer-review of framework code.

That's the plan so far. Now the questions:

1. Is it sensible? If not, any other ideas?
2. How does one implement this in perl? Using use Module we can only define the lowest version code is supposed to work with.

Answer 1

While I hope CPAN is more stable than the modules you're relying on, let me ask a similar question: how would you protect yourself against unexpected changes in a CPAN module?

One answer: you'd download the module and regress your code against it in a test environment.

Could the same be used here? Do you have to point to the "live" copies of their modules, or could you point to your own copies?

Answer 2

I would go about this by making a private copy of the libraries my code depends on and putting them in my project's lib directory with the understanding that I will never modify those copies except to periodically checkout new versions as milestones are reached.

Answer 3

It's a very sensible plan, and I implement it through a private CPAN-like repository that I've been calling 'DPAN'. You select the distributions and versions that you want from the real CPAN (or BackPAN), and create your own repository from it. Your CPAN clients point only at this repository, effectively freezing versions to exactly what you want. You only upgrade when you want to.

Additionally, the DPAN allows you to easily add not only your own local, private code, but to modify third party packages to fix problems with their installations, etc. I have a complete justification for the idea in the Summer 2009 issue of The Perl Review. You can also see my slides from my Creating Your Own CPAN talk at YAPC::Russia.

If you're interested in this sort of solution, check out my MyCPAN::App::DPAN module. It takes a directory of distros and does the rest for you. You point your CPAN client at it (and ensure that it won't connect to the internet) and that's that.

Once you can make your own repository, you can easily make a testing repository. Dump the versions that you think you want to upgrade into it, deploy the code on your testing server, and collect the results. If you don't like the results, you can easily change the repository.

The next big step in my DPAN work is to take an existing Perl installation, with whatever modules you might have installed, and create the repository that would give you that installation state. I have all the major pieces I need to do the work, but I've been a bit busy getting a couple of customers running with the first bits.

If you'd like to hear more about this stuff, just let me know. :)

Answer 4

Take a look at PAR. It lets you bundle up a set of dependencies into one file. You could take the modules they release, throw them in a PAR file and only upgrade the PAR file when you want to accept their changes.

Answer 5

If you want to check version of external modules, you could (if at least they report their $VERSION properly) use something like this :

BEGIN {
    use foo;
    use bar;

    die "Ghaaaa" if $foo::VERSION < 2.1;
    die "Aaaargh" if $foo::VERSION > 3.12;
}

Answer 6

Somebody pointed out PAR already. Let me mention PAR::Repository and it's companion module PAR::Repository::Client. They implement a client/server infrastructure that can automatically load any dependencies that aren't found locally (or that can even prefer the server's packages). As an administrator, you can simply add or remove packages to or from your repository. The actual serving of packages is done with utterly normal servers: Any http(s) server or simply file:// will do. Other protocols should be rather straightforward to implement.

It features the aforementioned magic auto-loading mechanism, package installation, and automatic package upgrading. Apart from the module documentation, you can have a look at a presentation on PAR from YAPC::Europe 2008 that covers this to some extend.

I have to admit that the automatic upgrading is sufficiently advanced technology that may eat a baby or two if it runs all out of kittens to gnaw on.