Portable command execution syntax implemented in Python

Question

Python is not a pretty good language in defining a set of commands to run. Bash is. But Bash does not run naively on Windows.

Background: I am trying to build a set of programs - with established dependency relationships between them - on mac/win/linux. Something like macports but should work on all the three platforms listed.

This begets the necessary to have a shell-like language that can be used to define the command sequence to run for building/patching/etc.. a particular program, for instance:

post-destroot {
    xinstall -d ${destroot}${prefix}/share/emacs/${version}/leim
    delete ${destroot}${prefix}/bin/ctags
    delete ${destroot}${prefix}/share/man/man1/ctags.1
    if {[variant_isset carbon]} {
        global version
        delete ${destroot}${prefix}/bin/emacs ${destroot}${prefix}/bin/emacs-${version}
    }
}

The above is from the emacs Portfile in macports. Note the use of variables, conditionals, functions .. besides specifying a simple list of commands to execute in that order.

Although the syntax is not Python, the actual execution semantics has to be deferred to Python using subprocess or whatever. In short, I should be able to 'run' such a script .. but for each command a specified hook function gets called that actually runs any command passed as argument. I hope you get the idea.

Answer 1

Sounds like you need some combination of PyParsing and Python Subprocess.

I find subprocess a little confusing, despite the MOTW about it, so I use this kind of wrapper code a lot.

from subprocess import Popen, PIPE

def shell(args, input=None):
    p = Popen(args, stdin=PIPE, stdout=PIPE, stderr=PIPE)
    stdout, stderr = p.communicate(input=input)
    return stdout, stderr

Answer 2

Ned Batchelder, in this post, lists several dozens of different tools you might choose to perform parsing, in Python, of some arbitrary language (though I don't quite understand why you don't want to use Python itself as your "language to define a set of commands to run", I'm sure you have your own reasons). One of these tools Ned lists is pyparsing, as Gregg mentions, but there are more than thirty others so you may want to have a look before you pick one that's to your taste.

Once you have transformed your input source language into a syntax tree or other convenient in-memory representation, you can just walk the tree and execute as you go (inserting variable values, branching on conditionals and loops, etc). Besides the ability to run external processes (e.g. via subprocess, whether wrapped up as Gregg suggests, or not), don't forget that Python itself may be able to execute some of your elementary commands without breaking a sweat, and, when feasible, that will be significantly faster than delegating execution to a child process (indeed, one early motivation for Perl's success was that it could do a lot of things within one process, while sh was forking away like mad; modern descendants of sh like bash and ksh took the lesson, and now implement a lot of built-ins that can execute within the same process as the main script;-).

For example, that delete command in your example can be implemented "internally" via os.unlink (can't link to the Python online docs right now as python.org is currently down due to HW problems;-).

Answer 3

Here's my trivial suggestion: parse it via regexp to Python so it becomes

def post_destroot():
    xinstall ('-d', '${destroot}${prefix}/share/emacs/${version}/leim')
    delete ('${destroot}${prefix}/bin/ctags')
    delete ('${destroot}${prefix}/share/man/man1/ctags.1')
    if test('variant_isset', 'carbon'):
        global('version')
        delete('${destroot}${prefix}/bin/emacs', '${destroot}${prefix}/bin/emacs-${version}')

I think it's not hard to write xinstall(), delete() and test() functions, especially since Python already has built-in function to format strings "{destroot}".format(dictionary).

But why bother? Try looking into distutils module from the standard library.