Using Linux redirect to overwrite file from Python script

Question

I have a simple python script that just takes in a filename, and spits out a modified version of that file. I would like to redirect stdout (using '>' from the command line) so that I can use my script to overwrite a file with my modifications, e.g. python myScript.py test.txt > test.txt

When I do this, the resulting test.txt does not contain any text from the original test.txt - just the additions made by myScript.py. However, if I don't redirect stdout, then the modifications come out correctly.

To be more specific, here's an example:

---

myScript.py:

#!/usr/bin/python
import sys

fileName = sys.argv[1]
sys.stderr.write('opening ' + fileName + '\n')

fileHandle = file(fileName)
currFile = fileHandle.read()
fileHandle.close()

sys.stdout.write('MODIFYING\n\n' + currFile + '\n\nMODIFIED!\n')

---

test.txt

Hello World

---

Result of python myScript.py test.txt > test.txt:

MODIFYING



MODIFIED!

Answer 1

The reason it works that way is that, before Python even starts, Bash interprets the redirection operator and opens an output stream to write stdout to the file. That operation truncates the file to size 0 - in other words, it clears the contents of the file. So by the time your Python script starts, it sees an empty input file.

The simplest solution is to redirect stdout to a different file, and then afterwards, rename it to the original filename.

python myScript.py test.txt > test.out && mv test.out test.txt

Alternatively, you could alter your Python script to write the modified data back to the file itself, so you wouldn't have to redirect standard output at all.

Answer 2

Try redirecting it to a new file, the redirect operator probably deletes the file just before appending.

Answer 3

The utility sponge present in the moreutils package in Debian can handle this gracefully.

python myScript.py test.txt | sponge test.txt

As indicated by its name, sponge will drain its standard input completely before opening test.txt and writing out the full contents of stdin.

You can get the latest version of sponge here. The moreutils home page is here.