How can I separate different sorts of Perl tests so I don't have to run them all?

Question

I noticed that in Perl the custom is to stick all tests into the t directory. How do you separate the unit test from the functional ones? Or, to make the question simpler and more obvious, how do you separate the tests that run quickly from the ones that do not? When all the tests run together the testing takes too long to be routinely used in development, which is a pity.

I figured I could set some environment variable like QUICK_TEST and skip the long tests according to its value. Do you separate unit and functional tests? How? (This is not meant to be a poll – I just thought maybe there’s some idiomatic solution.)

---

Update: So far I have come to this:

package Test::Slow;

use strict;
use Test::More;

BEGIN {
    plan(skip_all => 'Slow test.') if $ENV{QUICK_TEST};
}

1;

And in a nearby .t file:

# This is a slow test not meant
# to run frequently.
use Test::Slow;
use Test::More;

It seems to work nicely.

P.S. Now available as Test::Slow on CPAN.

Answer 1

You can certainly divide tests into subdirectories under t, with whatever categorization scheme you want. If you use an environment variable, I'd recommend making the default (if the variable is not set) be to run all tests. I've seen situations where t/ contains just the tests that would be routinely run in development and other tests are put under a different directory (e.g. t-selenium/).

I think it comes down to consistency being more important than which choice you make; just about anything will work if you are consistent.

Answer 2

Usually author-only test are put into xt directory. They are run manually. So, if you long tests are author-only, use xt. In general, t is common for CPAN modules. For private use, you can put them anywhere you like.

Answer 3

Run prove --state=all,save once to get some info added to .prove.

Run prove --state=slow -j9if you have a multi-core machine, and your tests can be run at the same time. This will cause your longest running tests to be started at the beginning, so that they will be more likely to finish before all of your other tests are done. This could reduce the overall time to completion, without preventing any tests from being run.

Answer 4

In Test::Manifest, I have a way to assign each test file a level. The test file only runs if the testing level is above that threshold. The lowe level would be the stuff I want to run all of the time, the next level the slightly slower stuff, and so on.

However, I hardly ever use it. If I'm concentrating on one part of a system, I just run the test for that part:

 % perl -Mblib t/some_test.t

Some people like to use prove to do the same thing.

I only end up running the full test suite when I need the integration testing to see if my changes broke anything else.