How do manage your Perl application development, build, and deployment?

Question

I have yet to come up with a satisfactory way to manage the development, build, and deployment of my Perl applications. I'd like to hear how you have solved this problem and/or what you would like to have in an application build system that you don't have now.

Please describe your application type (is it a web app, does it run on a server, or do you bundle it using PAR or PerlApp so that you can run on perlless systems).

Key things a build system should provide:

• Control of libraries.
  • It should be possible to check a library distribution out into my dev directory to use it in my build.
  • It should be easy to execute perl with an @INC value that will use the appropriate directories.
  • It should be possible to get a list of modules that are being sourced from the system perl install.
• Makefile/Build integration
  • It should be easy to do a global test across the entire application by issuing only one make test or similar command.
• Version control friendly
  • structure should not interfere with normal usage of CVS, SVN and other version control systems.
• Cross platform
  • System should operate on Win32 and Unix derived systems at minimum.
  • Ideally, tools should function identically in all places where perl operates.
• Single Perl install
  • It should not be necessary to install perl into a special directory as part of setting up the environment.
• Easy start up
  • Starting an application should be a mostly automated process. Something along the lines of Module::Starter or h2xs should be available to layout a basic structure and create any standard files.

Cross-posted at Perlmonks.

Answer 1

I work on a pretty small website application, and we're just working on improving our deployment (improving it from "spend a day setting up all the modules we need on Windows and then throw files at it until everything works", so that's some improvement).

We've got three things which we need doing to set up our website:

1. A Perl module made using Module::Starter, containing a Config module which holds the site-wide configuration options. On installation, this module (using MakeMaker's PREREQ_PM to check that all the modules we require have already been installed). Any modules which haven't need to be installed before this module can be installed.
2. A few SQL files which need to be executed to set up the database.
3. The Perl CGI files which make up the website. As long as Apache is pointed to them, the website "just works". This includes the common code modules used by all Perl files.

Deployment consists in me pulling from everybody's Git branches and packaging a version. We can then hand this over for testing, either locally or on an Amazon EC2 instance. Once we're good to release, we either install it over the last version, or move the database over to the testing instance and make that the new instance.

Comparing this to your criteria:

1. Control of libraries: Somewhat. We use CPAN modules quite extensively. To try a new version, we upgrade our own version of a module before doing that upgrade on the production server. We manually maintain a list, but since our codebase is fairly small, it's not hard to figure out which modules are being used (via greping for lines starting with use, for instance).
2. Makefile/Build integration: Yes. Any Makefile related stuff is done by our EU::MM setup. We don't have global tests, but since our entire test suite recently ended up in one folder, hopefully we'll soon have something you can run prove on directly.
3. Version control friendly: Yes. Our entire source code is contained in a single folder, without too much duplication.
4. Cross platform: Yes. We've got lots of weird stuff happening in MakeMaker to allow us to do this, but as a startup, having cross-platform code gives us valuable flexibility. We try to use Perl's core modules and tools, and Pure Perl modules from the CPAN, as much as possible.
5. Single Perl install: Yes. We can handle Perl being anywhere, and installed under any settings, as long as all of Perl's own module tools can work - there's been a lot of effort put into getting CPAN, EU::MM and others working well across all systems, and it seems a shame to waste it.
6. Easy start up: Not really. This system evolved (ie: was not intelligently designed) from a single folder of all the source files and a text file with a list of modules which need to be installed. While formalizing testing for installed modules is a huge improvement, it still takes us something like a day to set this up, mainly spent installing our prerequisite modules (not all of them are easy to install on Windows). I'm hoping to use the Perl Win32 community to try and get issues with problematic CPAN modules ironed out.

Mind you, it's a really simple website, no XS, complicated web framework, or any such. We've also only supported this setup through about two versions, so we don't have enough experience as to how this is going to work as the code gets more complicated and our deployment platforms become more varied. I'd really appreciate any suggestions or comments on our system.

Answer 2

There's a lot that I could write about this

1. Control of libraries - I create my own versions of CPAN with just the modules I want. The latest versions of App::Cpan has several features, such as the -j option to load one-time configurations, to help with this. Once you have this, you can distribute it on a thumb-drive or CD which has all of the modules, the CPAN.pm configuration, and everything else you need. With a little programming you create a rum_me script that make it all happen.

2. Makefile/Build integration - I don't integrate the Makefiles. That's the road to disaster. Instead, I do integration testing with the top-level application module, which automatically tests all of its dependencies too. The -t switch to the cpan command is useful to test the module in the current working directory:

   cpan -t .

There are various intergation testing frameworks that you can use too. You set PERL5LIB to something empty (with only have the core modules in the hard-coded @INC directories) so cpan has to install everything from scratch.

1. Version control friendly - it doesn't much matter what you use. Most things have some sort of export where you can get everything without the source control stuff. Git is very nice because it only has a minimum of pollution in the normal cases.

2. Cross platform - everything I've mentioned works just fine on Windows and Unix.

3. Single Perl install - this part is more tricky, and I think you're going in the wrong way. Any time multiple things have to depend on the same perl, someone is going to mess it up for the others. I definitely recommend not using the system Perl for application development just so you don't mess up the operation of the system. At the minimum, every application should install all non-core modules into their own directories so they don't compete with other applications.

4. Easy start up - that's just a simple matter of programming.

BONUS: I don't use Module::Starter. It's the wrong way to go since you have to depend what Module::Starter thinks you should do. I use Distribution::Cooker which simply takes a directory of Template Toolkit templates and processes them to give them your distribution directory. You can do anything that you like. How you get the initial templates is up to you.