Automated Testing of Binaries from SVN on HPC

Question

I would like to setup some automated testing of test cases upon Fortran binaries within an SVN Repository, on demand, or with a Cron job. To complicate the issue slightly, the test cases would be run on a computational cluster, so that each test case would need to generate a PBS script. (Linux Environment)

There are a lot of web testing, and unit testing solutions out there, however, I could not find anything for testing binaries directly. Say, provide inputs, and compare the outputs with the expected solution.

Any suggestions on how should this approached?

Answer 1

There may be a better answer that is more pre-packaged, but I believe that Buildbot is configurable enough that as long as you can write a python script to run your binary, it should work for what you want.

http://buildbot.net/

Answer 2

It seems like for simple input/output testing you could knock up a script yourself to do this...

e.g.

for each program
    compile
    submit to queue
    wait for finish
    check output

In practice, you'd probably want to submit more than one job to the queue, but the idea is the same.

Here are a few ideas off the top of my head on how you could do each step.

Submitting

You could use a template PBS script, and use sed to find/replace intput tokens with input values for the program before submitting

Waiting for finish

You could repeatedly grep the output of qstat for your job id, to wait for it to finish

Check output

Check the job's output file against expected output. You could either save the expected output to a file and use diff, or have a list of regular expression that must match for each run

Answer 3

I agree that this is something that would be pretty straightforward to script. Depending on how your binaries are configured and how different your programs are from one another, you could even include the testing scripts as part of your SVN repository.

Since you're in a batch system, you have some freedom for scheduling the tests. You may want to have a generic "verify" job that you can provide setup parameters to (e.g. to locations of expected and actual program output output). Nick mentioned using grep to check the output of qstat for your job ID, but you can tell PBS to hold a job until another job completes. This would mean that you have something like:

...
#PBS -N run_test_1
#PBS -a 200906270000
...
<compile test>
<run test>
<save output>

when submitted, save the job ID returned by qsub (how you do this is dependent on your platform - usually something like job_id=$(qsub $JOB_FILE) is sufficient). Then, plug that value in to another script:

...
#PBS -N verify_test_1
#PBS -W depend=afterany:$job_id
<run comparison>
...

This will (when the proper value of job_id is inserted) hold the execution of the test run until midnight on June 27, 2009, and hold the execution of the verification job until the test job completes (the afterany directive indicates that it should always run after the first job - not just if it's successful).

Depending on your cluster's turnaround time, though, you could put all of this in one script, though you could still use the PBS time-based holds to only run at a particular time. I've recently started using Python instead of shell scripts to handle even this system-related jobs - I tested yesterday making the Python script executable and adding the PBS directives straight into the source - it seemed to work very well.