How to make shell output redirect (>) write while script is still running?

Question

I wrote a short script that never terminates. This script continuously generates output that I have to check on every now and then. I'm running it on a lab computer through SSH, and redirecting the output to a file in my public_html folder on that machine.

python script.py > ~/public_html/results.txt

However, the results don't show up immediately when I refresh the address. The results show up when I terminate the program, but as I said, it doesn't halt by itself. Is that redirect (>) being lazy with with writing? Is there a way to continuously (or with an interval) update the results in the file?

Or is it the webserver that doesn't update the file while it is still being written?

Answer 1

You need to flush the output sys.stdout.flush() (or smth) if you want to see it immediately. See this

Answer 2

I suspect the file is being continuously written, but that the web server is reporting the modified date of the file as the time it was opened, and thus is reporting that no change to the file has occurred and the result is being cached (either at the web server or at the client).

I would first try a forced reload (Ctrl+F5 or Ctrl+Shift+R or Shift+<reload_button>) and see if that helps. If it doesn't, then you can try something else.

In a separate shell on the server, do

tail -f ~/public_html/results.txt

Tail prints out the last n lines of the file (where n defaults to 10), but the -f parameter monitors the file and continues to report output as the file grows. This will at least give you confidence that the file is being written to incrementally.

I hope that helps.

Answer 3

stdout is buffered, if not connected to terminal.

You can change this policy to line-buffering via stdbuf

stdbuf -oL python script.py > ~/public_html/results.txt

So you don't have to flush in your Python script and keep it IO efficient, if line-buffering is not required.