Running doctests through iPython and pseudo-consoles

Question

I've got a fairly basic doctestable file:

class Foo():
    """
    >>> 3+2
    5
    """

if __name__ in ("__main__", "__console__"):
    import doctest
    doctest.testmod(verbose=True)

which works as expected when run directly through python.

However, in iPython, I get

1 items had no tests:
    __main__
0 tests in 1 items.
0 passed and 0 failed.
Test passed.

Since this is part of a Django project and will need access to all of the appropriate variables and such that manage.py sets up, I can also run it through a modified command, which uses code.InteractiveConsole, one result of which is __name__ gets set to '__console__'.

With the code above, I get the same result as with iPython. I tried changing the last line to this:

 this = __import__(__name__)
 doctest.testmod(this, verbose=True)

and I get an ImportError on __console__, which makes sense, I guess. This has no effect on either python or ipython.

So, I'd like to be able to run doctests successfully through all three of these methods, especially the InteractiveConsole one, since I expect to be needing Django pony magic fairly soon.

Just for clarification, this is what I'm expecting:

Trying:
    3+2
Expecting:
    5
ok
1 items had no tests:
    __main__
1 items passed all tests:
   1 tests in __main__.Foo
1 tests in 2 items.
1 passed and 0 failed.
Test passed.

Answer 1

The following works:

$ ipython
...
In [1]: %run file.py

Trying:
    3+2
Expecting:
    5
ok
1 items had no tests:
    __main__
1 items passed all tests:
   1 tests in __main__.Foo
1 tests in 2 items.
1 passed and 0 failed.
Test passed.

In [2]:

I have no idea why ipython file.py does not work. But the above is at least a workaround.

EDIT:

I found the reason why it does not work. It is quite simple:

• If you do not specify the module to test in doctest.testmod(), it assumes that you want to test the __main__ module.
• When IPython executes the file passed to it on the command line, the __main__ module is IPython's __main__, not your module. So doctest tries to execute doctests in IPython's entry script.

The following works, but feels a bit weird:

if __name__ == '__main__':
    import doctest
    import the_current_module
    doctest.testmod(the_current_module)

So basically the module imports itself (that's the "feels a bit weird" part). But it works. Something I do not like abt. this approach is that every module needs to include its own name in the source.

EDIT 2:

The following script, ipython_doctest, makes ipython behave the way you want:

#! /usr/bin/env bash

echo "__IP.magic_run(\"$1\")" > __ipython_run.py
ipython __ipython_run.py

The script creates a python script that will execute %run argname in IPython.

Example:

$ ./ipython_doctest file.py
Trying:
    3+2
Expecting:
    5
ok
1 items had no tests:
    __main__
1 items passed all tests:
   1 tests in __main__.Foo
1 tests in 2 items.
1 passed and 0 failed.
Test passed.
Python 2.5 (r25:51908, Mar  7 2008, 03:27:42) 
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]:

Answer 2

The root problem is that ipython plays weird tricks with __main__ (through its own FakeModule module) so that, by the time doctest is introspecting that "alleged module" through its __dict__, Foo is NOT there -- so doctest doesn't recurse into it.

Here's one solution:

class Foo():
    """
    >>> 3+2
    5
    """

if __name__ in ("__main__", "__console__"):
    import doctest, inspect, sys
    m = sys.modules['__main__']
    m.__test__ = dict((n,v) for (n,v) in globals().items()
                            if inspect.isclass(v))
    doctest.testmod(verbose=True)

This DOES produce, as requested:

$ ipython dot.py 
Trying:
    3+2
Expecting:
    5
ok
1 items had no tests:
    __main__
1 items passed all tests:
   1 tests in __main__.__test__.Foo
1 tests in 2 items.
1 passed and 0 failed.
Test passed.
Python 2.5.1 (r251:54863, Feb  6 2009, 19:02:12) 
  [[ snip snip ]]
In [1]:

Just setting global __test__ doesn't work, again because setting it as a global of what you're thinking of as __main__ does NOT actually place it in the __dict__ of the actual object that gets recovered by m = sys.modules['__main__'], and the latter is exactly the expression doctest is using internally (actually it uses sys.modules.get, but the extra precaution is not necessary here since we do know that __main__ exists in sys.modules... it's just NOT the object you expect it to be!-).

Also, just setting m.__test__ = globals() directly does not work either, for a different reason: doctest checks that the values in __test__ are strings, functions, classes, or modules, and without some selection you cannot guarantee that globals() will satisfy that condition (in fact it won't). Here I'm selecting just classes, if you also want functions or whatnot you can use an or in the if clause in the genexp within the dict call.

I don't know exactly how you're running a Django shell that's able to execute your script (as I believe python manage.py shell doesn't accept arguments, you must be doing something else, and I can't guess exactly what!-), but a similar approach should help (whether your Django shell is using ipython, the default when available, or plain Python): appropriately setting __test__ in the object you obtain as sys.modules['__main__'] (or __console__, if that's what you're then passing on to doctest.testmod, I guess) should work, as it mimics what doctest will then be doing internally to locate your test strings.

And, to conclude, a philosophical reflection on design, architecture, simplicity, transparency, and "black magic"...:

All of this effort is basically what's needed to defeat the "black magic" that ipython (and maybe Django, though it may be simply delegating that part to ipython) is doing on your behalf for your "convenience"... any time at which two frameworks (or more;-) are independently doing each its own brand of black magic, interoperability may suddenly require substantial effort and become anything BUT convenient;-).

I'm not saying that the same convenience could have been provided (by any one or more of ipython, django and/or doctests) without black magic, introspection, fake modules, and so on; the designers and maintainers of each of those frameworks are superb engineers, and I expect they've done their homework thoroughly, and are performing only the minimum amount of black magic that's indispensable to deliver the amount of user convenience they decided they needed. Nevertheless, even in such a situation, "black magic" suddenly turns from a dream of convenience to a nightmare of debugging as soon as you want to do something even marginally outside what the framework's author had conceived.

OK, maybe in this case not quite a nightmare, but I do notice that this question has been open a while and even with the lure of the bounty it didn't get many answers yet -- though you now do have two answers to pick from, mine using the __test__ special feature of doctest, @codeape's using the peculiar __IP.magic_run feature of ironpython. I prefer mine because it does not rely on anything internal or undocumented -- __test__ IS a documented feature of doctest, while __IP, with those two looming leading underscores, scream "deep internals, don't touch" to me;-)... if it breaks at the next point release I wouldn't be at all surprised. Still, matter of taste -- that answer may arguably be considered more "convenient".

But, this is exactly my point: convenience may come at an enormous price in terms of giving up simplicity, transparency, and/or avoidance of internal/undocumented/unstable features; so, as a lesson for all of us, the least black magic &c we can get away with (even at the price of giving up an epsilon of convenience here and there), the happier we'll all be in the long run (and the happier we'll make other developers that need to leverage our current efforts in the future).