testing command line utilities

Question

I'm looking for a way to run tests on command-line utilities written in bash, or any other language.

I'd like to find a testing framework that would have statements like

setup:
    command = 'do_awesome_thing'
    filename = 'testfile'
    args = ['--with', 'extra_win', '--file', filename]
    run_command command args

test_output_was_correct
    assert_output_was 'Creating awesome file "' + filename + '" with extra win.'

test_file_contains_extra_win
    assert_file_contains filename 'extra win'

Presumably the base test case would set up a temp directory in which to run these commands, and remove it at teardown.

I would prefer to use something in Python, since I'm much more familiar with it than with other plausible candidate languages.

I imagine that there could be something using a DSL that would make it effectively language-agnostic (or its own language, depending on how you look at it); however this might be less than ideal, since my testing techniques usually involve writing code that generates tests.

It's a bit difficult to google for this, as there is a lot of information on utilities which run tests, which is sort of the converse of what I'm looking for.

Support for doctests embedded in the output of command --help would be an extra bonus :)

Answer 1

Well... What we usually do (and one of the wonders of O.O. languages) is to write all the components of an application before actually make the application. Every component might have an standalone way to be executed, for testing purpose (command line, usually), that also allows you to think in them as complete programs each by each, and use them in future projects. If what you want is to test the integrity of an existing program... well, I think the best way is to learn in deep how it work, or even deeper: read the source. Or even deeper: develop a bot to force-test it :3

Sorry that's what I have .-.

Answer 2

Check out ScriptTest :

from scripttest import TestFileEnvironment

env = TestFileEnvironment('./scratch')

def test_script():
    env.reset()
    result = env.run('do_awesome_thing testfile --with extra_win --file %s' % filename)
    # or use a list like ['do_awesome_thing', 'testfile', ...]
    assert result.stdout.startswith('Creating awesome file')
    assert filename in result.files_created

It's reasonably doctest-usable as well.

Answer 3

outside of any prepackaged testing framework that may exist but I am unaware of, I would just point out that expect is an awesome and so underutilized tool for this kind of automation, especially if you want to support multistage interaction, which is to say not just send a command and check output but respond to output with more input. If you wind up building your own system, it's worth looking into.

There is also python reimplementation of expect called pexpect.There may be some direct interfaces to the expect library available as well. I'm not a python guy so I couldn't tell you much about them.