Python print statements being buffered with > output redirection

Question

I'm doing print statements in python. I'm executing my script like so:

python script.py > out.log nohup &

The print statements are not all showing up in out.log but the program is finishing ok.

That line of code is in an .sh file I execute by doing ./script.sh

Update: The log does get all the data but not until a certain # of lines are printed. It would appear to be buffering the output.

Answer 1

Did you try:

1. Running it without the redirection and making sure everything is printed as expected
2. Running it not through a shell script, but simply from a command line

?

These steps may help you "home in" on the problem.

Answer 2

When stdout is sent to a tty it will be line buffered and will be flushed every line, but when redirected to a file or pipe it'll be fully buffered and will only be flushed periodically when you overrun the buffer.

You'll have to add sys.stdout.flush() calls after each line if you want the output to be immediately visible, or disable buffering entirely. See http://stackoverflow.com/questions/107705/python-output-buffering for ways to do the latter:

Suggestions so far:

1. Use the -u command line switch
2. Wrap sys.stdout in an object that flushes after every write
3. Set PYTHONUNBUFFERED env var
4. sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)