Ouput file redirection in Python

Question

I'm writing a backup script I intend to execute in a cronjob every night.

The script sets sys.stdout and sys.stderr to an output file to keep a log of what happens.

To do the backup I use the following code

cmd = 'rsync -av --del --stats --filter "- .thumbnails/" ' + \
    '--filter "- *~" --filter "- *.iso" --filter "- lost+found/" ' + \
    '--filter "- .cache/" --filter "- tmp/" --filter "- *.mp3" ' + \
    '--filter "- *.log" ' + srcDir + ' ' + dstDir

print "Executing '"+cmd+"' ..."
try:
    sys.stdout.flush()
    sys.stderr.flush()
    retcode = subprocess.call( cmd, stdin = sys.stdin, stdout = sys.stdout,
        stderr=sys.stderr, shell=False )
    if retcode < 0:
        print >>sys.stderr, "Command was terminated by signal", -retcode
    elif retcode > 0:
        print >>sys.stderr, "Command returned code ", retcode
except OSError, e:
    print >>sys.stderr, "Execution failed:", e

I add print statements before and after the subprocess call. The problem is that I get the output of the subprocess call before any output of my print instructions before the call. I added the flush() calls but it has no effect.

Why is this happening and how could I change this behaviour ?

Answer 1

Have you tried putting the flush calls outside the try block?

Answer 2

Why are you printing to stderr? If the subprocess is writing to stdout while you are writing to stderr, that could explain the odd interleaving.

Answer 3

I just found the solution here in a Stackoverflow answer.

Replace

sys.stderr = sys.stdout = logFile = open( tmpLogFileName, 'a' )

with

sys.stderr = sys.stdout = logFile = open( tmpLogFileName, 'a', 0 )

This tells python to not assign any output buffer to file.