Linux: Pipe into Python (ncurses) script, stdin and termios

Question

Hi all,

Apparently this is almost a duplicate of "Bad pipe filedescriptor when reading from stdin in python - Stack Overflow"; however, I believe this case is slightly more complicated (and it is not Windows specific, as the conclusion of that thread was).

I'm currently trying to experiment with a simple script in Python: I'd like to supply input to the script - either through command line arguments; or by 'pipe'-ing a string to this script - and have the script show this input string using a curses terminal interface.

The full script, here called testcurses.py, is given below. The problem is that whenever I try the actual piping, that seems to mess up stdin, and the curses window never shows. Here is a terminal output:

## CASE 1: THROUGH COMMAND LINE ARGUMENT (arg being stdin):
##
$ ./testcurses.py -
['-'] 1
stdout/stdin (obj): <open file '<stdout>', mode 'w' at 0xb77dc078> <open file '<stdin>', mode 'r' at 0xb77dc020>
stdout/stdin (fn): 1 0
env(TERM): xterm xterm
stdin_termios_attr [27906, 5, 1215, 35387, 15, 15, ['\x03', ... '\x00']]
stdout_termios_attr [27906, 5, 1215, 35387, 15, 15, ['\x03', ... '\x00']]
opening -
obj <open file '<stdin>', mode 'r' at 0xb77dc020>
TYPING blabla HERE
wr TYPING blabla HERE

at end
before curses TYPING blabla HERE
#
# AT THIS POINT:
# in this case, curses window is shown, with the text 'TYPING blabla HERE'
# ################


## CASE 2: THROUGH PIPE
##
## NOTE I get the same output, even if I try syntax as in SO1057638, like:
## python -c "print 'TYPING blabla HERE'" | python testcurses.py -
##
$ echo "TYPING blabla HERE" | ./testcurses.py -
['-'] 1
stdout/stdin (obj): <open file '<stdout>', mode 'w' at 0xb774a078> <open file '<stdin>', mode 'r' at 0xb774a020>
stdout/stdin (fn): 1 0
env(TERM): xterm xterm
stdin_termios_attr <class 'termios.error'>::(22, 'Invalid argument')
stdout_termios_attr [27906, 5, 1215, 35387, 15, 15, ['\x03', '\x1c', '\x7f', '\x15', '\x04', '\x00', '\x01', '\xff', '\x11', '\x13', '\x1a', '\xff', '\x12', '\x0f', '\x17', '\x16', '\xff', '\x00', '\x00', '\x00', '\x00', '\x00', '\x00', '\x00', '\x00', '\x00', '\x00', '\x00', '\x00', '\x00', '\x00', '\x00']]
opening -
obj <open file '<stdin>', mode 'r' at 0xb774a020>
wr TYPING blabla HERE

at end
before curses TYPING blabla HERE
#
# AT THIS POINT:
# script simply exits, nothing is shown 
# ################

As far as I can see, the issue is: - whenever we pipe strings into the Python script, the Python script loses the reference to the terminal as stdin, and notices that the replaced stdin is not a termios structure anymore - and since stdin is no longer a terminal, curses.initscr() exits immediately without rendering anything.

So, my question is - in brief: can I somehow achieve, that the syntax echo "blabla" | ./testcurses.py - ends up showing the piped string in curses? More specifically: is it possible to retrieve a reference to the calling terminal's stdin from a Python script, even if this script is being "piped" to?

Thanks in advance for any pointers,

Cheers!

 

 

PS: the testcurses.py script:

#!/usr/bin/env python 
# http://www.tuxradar.com/content/code-project-build-ncurses-ui-python
# http://diveintopython.org/scripts_and_streams/stdin_stdout_stderr.html
# http://bytes.com/topic/python/answers/42283-curses-disable-readline-replace-stdin
#
# NOTE: press 'q' to exit curses - Ctrl-C will screw up yer terminal

# ./testcurses.py "blabla"                  # works fine (curseswin shows)
# ./testcurses.py -                     # works fine, (type, enter, curseswins shows):
# echo "blabla" | ./testcurses.py "sdsd"        # fails to raise curses window 
# 
# NOTE: when without pipe: termios.tcgetattr(sys.__stdin__.fileno()): [27906, 5, 1215, 35387, 15, 15, ['\x03', 
# NOTE: when with pipe |   : termios.tcgetattr(sys.__stdin__.fileno()): termios.error: (22, 'Invalid argument') 

import curses
import sys
import os
import atexit
import termios

def openAnything(source):            
    """URI, filename, or string --> stream

    http://diveintopython.org/xml_processing/index.html#kgp.divein

    This function lets you define parsers that take any input source
    (URL, pathname to local or network file, or actual data as a string)
    and deal with it in a uniform manner.  Returned object is guaranteed
    to have all the basic stdio read methods (read, readline, readlines).
    Just .close() the object when you're done with it.
    """
    if hasattr(source, "read"):
        return source

    if source == '-':
        import sys
        return sys.stdin

    # try to open with urllib (if source is http, ftp, or file URL)
    import urllib                         
    try:                                  
        return urllib.urlopen(source)     
    except (IOError, OSError):            
        pass                              

    # try to open with native open function (if source is pathname)
    try:                                  
        return open(source)               
    except (IOError, OSError):            
        pass                              

    # treat source as string
    import StringIO                       
    return StringIO.StringIO(str(source)) 



def main(argv):

    print argv, len(argv)
    print "stdout/stdin (obj):", sys.__stdout__, sys.__stdin__ 
    print "stdout/stdin (fn):", sys.__stdout__.fileno(), sys.__stdin__.fileno()
    print "env(TERM):", os.environ.get('TERM'), os.environ.get("TERM", "unknown")

    stdin_term_attr = 0
    stdout_term_attr = 0
    try:
        stdin_term_attr = termios.tcgetattr(sys.__stdin__.fileno())
    except:
        stdin_term_attr = "%s::%s" % (sys.exc_info()[0], sys.exc_info()[1]) 
    try:
        stdout_term_attr = termios.tcgetattr(sys.__stdout__.fileno())
    except:
        stdout_term_attr = `sys.exc_info()[0]` + "::" + `sys.exc_info()[1]` 
    print "stdin_termios_attr", stdin_term_attr
    print "stdout_termios_attr", stdout_term_attr


    fname = ""
    if len(argv):
        fname = argv[0]

    writetxt = "Python curses in action!"
    if fname != "":
        print "opening", fname
        fobj = openAnything(fname)
        print "obj", fobj
        writetxt = fobj.readline(100) # max 100 chars read
        print "wr", writetxt
        fobj.close()
        print "at end"

    sys.stderr.write("before ")
    print "curses", writetxt
    try:
        myscreen = curses.initscr()
        #~ atexit.register(curses.endwin)
    except:
        print "Unexpected error:", sys.exc_info()[0]

    sys.stderr.write("after initscr") # this won't show, even if curseswin runs fine

    myscreen.border(0)
    myscreen.addstr(12, 25, writetxt)
    myscreen.refresh()
    myscreen.getch()

    #~ curses.endwin()
    atexit.register(curses.endwin)

    sys.stderr.write("after end") # this won't show, even if curseswin runs fine


# run the main function - with arguments passed to script:
if __name__ == "__main__":
    main(sys.argv[1:])
    sys.stderr.write("after main1") # these won't show either, 
sys.stderr.write("after main2")     #  (.. even if curseswin runs fine ..)

Answer 1

This can't be done without getting the parent process involved. Fortunately, there's a way to get bash involved using I/O redirection :

$ (echo "foo" | ./pipe.py) 3<&0

That will pipe foo to pipe.py in a subshell with stdin duplicated into file descriptor 3. Now all we need to do is use that extra help from our parent process in the python script (since we'll inherit fd 3):

#!/usr/bin/env python

import sys, os
import curses

output = sys.stdin.readline(100)

# We're finished with stdin. Duplicate inherited fd 3,
# which contains a duplicate of the parent process' stdin,
# into our stdin, at the OS level (assigning os.fdopen(3)
# to sys.stdin or sys.__stdin__ does not work).
os.dup2(3, 0)

# Now curses can initialize.
screen = curses.initscr()
screen.border(0)
screen.addstr(12, 25, output)
screen.refresh()
screen.getch()
curses.endwin()

Finally, you can work around the ugly syntax on the command line by running the subshell first:

$ exec 3<&0  # spawn subshell
$ echo "foo" | ./pipe.py  # works
$ echo "bar" | ./pipe.py  # still works

That solves your problem, if you have bash.

Answer 2

The problem is that whenever I try the actual piping, that seems to mess up stdin, and the curses window never shows. [...snip...] As far as I can see, the issue is: - whenever we pipe strings into the Python script, the Python script loses the reference to the terminal as stdin, and notices that the replaced stdin is not a termios structure anymore - and since stdin is no longer a terminal, curses.initscr() exits immediately without rendering anything.

Actually, the curses window does show, but since there is no more input on your brave new stdin, myscreen.getch() returns immediately. So it has nothing to do with curses testing whether stdin is a terminal.

So if you want to use myscreen.getch() and other curses input functions, you'll have to reopen your terminal. On Linux and *nix systems there is usually a device called /dev/tty that refers to the current terminal. So you can do something like:

f=open("/dev/tty")
os.dup2(f.fileno(), 0)

before your call to myscreen.getch().