How do I go about setting up a TDD development process with Google App Engine?

Question

I'm primarily a Ruby guy, but lately I've been working on a lot of Python stuff, in particular, App Engine code. In Ruby, I'd use automated continuous integration (autotest), code coverage tools (rcov), static analysis (reek), and mutation testing (heckle) in my development process, but I'm not sure how best to set up a similar development process for an App Engine environment. I'd also be interested in analogs to RSpec and Cucumber for Python that could work in App Engine.

Answer 1

You won't always find one to one equivalents of Ruby testing tools in Python, but there are some great testing tools in Python. Some of the tools that I've found useful include:

• unittest - the xUnit tool included in the Python standard library. It includes all the basics for unit testing.
• doctest - an awesome part of the standard library, it allows you to write tests in the docstrings of functions, classes, modules, methods. It is great at conveying intended API usage. Ian Bicking suggests using doctest for Behavior Driven Development. Doctest fits very well into the Sphinx documentation system (you can make sure all the examples in your documentation pass every time you build the docs).
• nose and py.test are seen as the next-gen versions of unittest. They can run all existing unittest cases, but allow for easier, non-class based unit tests. py.test also allows for distributed execution.
• mock is a nice library for mocking behavior.
• tdaemon watches a directory for updates to your code and will re-execute your test suite. (my personal branch contains a few unmerged improvements).
• Buildbot, Bitten, and even Hudson all work well as full-fledged continuous integration servers for Python code.
• coverage.py computes the code coverage of your code.
• pylint will provide a lint-like analysis of your code, making sure it follows common coding conventions and does not have any common bugs. There is also a "lighter" analysis tool, PyFlakes.
• There are a number of HTTP / Browser testing tools that work well in Python, including Twill, Selenium, and Windmill.

If you are using Django on App Engine, it includes several extensions to unittest that allow you simulate an HTTP client and database persistence.

There are a ton of other tools that I have not used (such as PySpec and Behaviour) that could also be helpful. I haven't seen any mutation testing tool in Python, but I bet there is one out there (I would love to learn what it is).

Happy testing!

Answer 2

Haven't used App Engine, but my feeling for the most popular python testing tools is

• unittest/doctest are the testing packages from the Python standard library. unittest is the xUnit for python.
• nose is a test runner/finder. It has many options, including --with-coverage, which uses coverage to give you code coverage reports.
• pylint is the most featureful lint-checker for python. Useful beyond a syntax checker since it advises on unused variables/functions, when methods should be functions, and more.
• pester (mutation testing)
• buildbot (continuous integration)

You'll probably want to reference this (not quite complete) list of Python Testing Tools.

For BDD, the field was thin last time I checked. Many of the true BDD tools were not usable with nose and/or too limiting in the syntax they required. You might have some luck with spec, which is a BDD-like nose plugin. Just found pyccuracy, which looks a lot like cucumber, but I haven't tried it.

For what its worth, I now just use nosetests -v (the nose runner with --verbose), which will use the first line of the docstring in the test runner output. That is, given a test like:

class TestFoo(unittest.TestCase):
    def testAnyNameHere(self):
        """ Foo should be bar"""
        foo = "bar"
        self.assertEqual(foo, 'bar')

nosetests will give:

$ nosetests -v
Foo should be bar... ok

-----------------------------
Ran 1 tests in 0.002s
OK