Python output buffering

Question

Is output buffering enabled by default in Python's interpreter for sys.stdout ?

If the answer is positive, what are all the ways to disable it ?

Suggestions so far:

1. Use the -u command line switch
2. Wrap sys.stdout in an object that flushes after every write
3. Set PYTHONUNBUFFERED env var
4. sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

Is there any other way to set some global flag in sys / sys.stdout programmatically during execution ?

Answer 1

From Magnus Lycka answer on a mailing list:

You can skip buffering for a whole python process using "python -u" (or#!/usr/bin/env python -u etc) or by setting the environment variable PYTHONUNBUFFERED.

You could also replace sys.stdout with some other stream like wrapper which does a flush after every call.

>>> class Unbuffered:
..     def __init__(self, stream):
..         self.stream = stream
..     def write(self, data):
..         self.stream.write(data)
..         self.stream.flush()
..     def __getattr__(self, attr):
..         return getattr(self.stream, attr)
..
>>> import sys
>>> sys.stdout=Unbuffered(sys.stdout)
>>> print 'Hello'

Answer 2

Yes, it is.

You can disable it on the commandline with the "-u" switch.

Alternatively, you could call .flush() on sys.stdout on every write (or wrap it with an object that does this automatically)

Answer 3

Yes, it is enabled by default. You can disable it by using the -u option on the command line when calling python.

Answer 4

One way to get unbuffered output would be to use sys.stderr instead of sys.stdout or to simply call sys.stdout.flush() to explicitly force a write to occur.

You could easily redirect everything printed by doing:

import sys; sys.stdout = sys.stderr

print "Hello World!"

Or to redirect just for a particular print statement:

print >>sys.stderr, "Hello World!"

To reset stdout you can just do:

sys.stdout = sys.__stdout__

Answer 5

You can create an unbuffered file and assign this file to sys.stdout.

import sys 
myFile= open( "a.log", "w", 0 ) 
sys.stdout= myFile

You can't magically change the system-supplied stdout; since it's supplied to your python program by the OS.

Answer 6

# reopen stdout file descriptor with write mode
# and 0 as the buffer size (unbuffered)
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

Credits: "Sebastian", somewhere on the Python mailing list.

Answer 7

You can also use fcntl to change the file flags in-fly.

fl = fcntl.fcntl(fd.fileno(), fcntl.F_GETFL)
fl |= os.O_SYNC # or os.O_DSYNC (if you don't care the file timestamp updates)
fcntl.fcntl(fd.fileno(), fcntl.F_SETFL, fl)

Answer 8

def disable_stdout_buffering():
    # Appending to gc.garbage is a way to stop an object from being
    # destroyed.  If the old sys.stdout is ever collected, it will
    # close() stdout, which is not good.
    gc.garbage.append(sys.stdout)
    sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

# Then this will give output in the correct order:
disable_stdout_buffering()
print "hello"
subprocess.call(["echo", "bye"])

Without saving the old sys.stdout, disable_stdout_buffering() isn't idempotent, and multiple calls will result in an error like this:

Traceback (most recent call last):
  File "test/buffering.py", line 17, in <module>
    print "hello"
IOError: [Errno 9] Bad file descriptor
close failed: [Errno 9] Bad file descriptor

Another possibility is:

def disable_stdout_buffering():
    fileno = sys.stdout.fileno()
    temp_fd = os.dup(fileno)
    sys.stdout.close()
    os.dup2(temp_fd, fileno)
    os.close(temp_fd)
    sys.stdout = os.fdopen(fileno, "w", 0)

(Appending to gc.garbage is not such a good idea because it's where unfreeable cycles get put, and you might want to check for those.)