Why does bash sometime not flush output of a python program to a file

Question

I have a crontab job calling a python script and outputting to a file:

python run.py &> current_date.log

now sometimes when I do

tail -f current_date.log

I see the file filling up with the output, but other times the log file exists, but stays empty for a long time. I am sure that the python script is printing stuff right after it starts running, and the log file is created. Any ideas why does it stay empty some of the time?

Answer 1

The problem is actually Python (not bash) and is by design. Python buffers output by default. Run python with -u to prevent buffering.

Another suggestion is to create a class (or special function) which calls flush() right after the write to the log file.

Answer 2

Python buffers output when it detects that it is not writing to a tty, and so your log file may not receive any output right away. You can configure your script to flush output or you can invoke python with the -u argument to get unbuffered output.

$ python -h

...

-u     : unbuffered binary stdout and stderr (also PYTHONUNBUFFERED=x)
         see man page for details on internal buffering relating to '-u'

...