Why does my python interactive console not work properly?

Question

I made a very simple interactive console that I'd like to use in a complicated scraping application. It looks like this:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import os, sys, codecs, code
sys.__stdout__ = codecs.getwriter('utf8')(sys.__stdout__)
sys.__stderr__ = codecs.getwriter('utf8')(sys.__stderr__)

if 'DEBUG' in os.environ:
    import pdb 
    import sys 
    oeh = sys.excepthook
    def debug_exceptions(type, value, traceback):
        pdb.post_mortem(traceback)
        oeh(type, value, traceback)
    sys.excepthook = debug_exceptions

class CLI(code.InteractiveConsole):
    def __init__(self, locals=None, filename="<console>", histfile=None):
        code.InteractiveConsole.__init__(self, locals, filename)
        try:
            import readline
        except ImportError:
            pass
        else:
            try:
                import rlcompleter
                readline.set_completer(rlcompleter.Completer(locals).complete)
            except ImportError:
                pass
            readline.parse_and_bind("tab: complete")
        self.interact()

if __name__ == "__main__":
    hello="I am a local"
    CLI(locals=locals())

If I call it from another simple application, it works just fine:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import os, sys, codecs, cli
sys.__stdout__ = codecs.getwriter('utf8')(sys.__stdout__)
sys.__stderr__ = codecs.getwriter('utf8')(sys.__stderr__)

from cli import CLI

foo="i am a local"
CLI(locals=locals())

However, when I call it from my scraping framework which is based off twill and mechanize for now (though I intend to switch it to gevent) when call up the CLI in exactly the same way, the arrow keys don't work, tab completion doesn't work, in fact it behaves like readline doesn't exist. I've tried reloading the readline module and passing it direct parse_and_bind commands but for some reason it just will not play properly. Any hints or suggestions as to what has been clobbered that is preventing it from working as expected or am I just going to have to remove all the external modules in use and put them in, one by one to see what happened?

I'm suspicious of twill seeing as it has it's own basic CLI but if anyone knows I'd be very happy to know if anyone has a good idea what's going on.

Oh and please no comments about what I'm doing with stderr and stdout, it's just boilerplate code that gets put into python files, I always run them from utf8 consoles and it's not what I'm asking about...

Answer 1

OK, I found out it was ME that was causing the problem, my older boiler-plate code used to this before I noticed it causing problems in some cases:

sys.stdout = codecs.getwriter('utf8')(sys.stdout)
sys.stderr = codecs.getwriter('utf8')(sys.stderr)

That old code was still present in one of my own files, changing that code to the lower-level version of:

sys.__stdout__ = codecs.getwriter('utf8')(sys.__stdout__)
sys.__stderr__ = codecs.getwriter('utf8')(sys.__stderr__)

Or removing it completely, since it didn't need to be in that file anyway fixed the issue.