Line-by-line Remote Data Transmission in Python

Question

I've been playing with the subprocess module to iteratively send each line in an input file to a process created by the following command.

ssh -t -A $host 'remote_command'

The remote_command expects a line in its STDIN, does some processing on the line and iterates the cycle until STDIN closes or reaches EOF.

To achieve this, what I'd been doing was:

process = subprocess.Popen("ssh -t -A $host 'remote_command'",
                           shell=True,
                           stdin=subprocess.PIPE)
for line in file('/tmp/foo'):
  process.stdin.write(line)
  process.stdin.flush()
process.stdin.close()

But what I discovered was that the above method is not robust enough, as it is often the case that remote_command finishes prematurely without processing the entire content (though sometimes the same code does succeed without a problem).

The situation is the same when I employ another, albeit very simiar, approach:

process = subprocess.Popen("ssh -t -A $host 'remote_command'",
                           shell=True,
                           stdin=file('/tmp/foo'))

So the question is: How can I make sure that each line in an input file be sent, received, and processed until the end by the remote machine in Python?

Answer 1

Instead of wrapping around a subprocess, you'd likely be better off using something like paramiko.

But in either case, if your connection gets terminated before you've sent all your data, you can catch that exception and you'll know that you need to retry. In the event the process dies prematurely you should be able to read the exit-code of the process.

Answer 2

You can't do much more than what you have done to make sure that all input gets sent to your child process. Your second example is better than the first one, in my opinion. What you can do is inspecting the return code from your child process.

return_code = p.wait()

Your remote command should probably return 0 on successful completion and something non zero if an error occurred.