Sending large file to PIPE input in python

Question

I have the following code:

sourcefile = open(filein, "r")
targetfile = open(pathout, "w")

content= sourcefile.read():

p = Popen([SCRIPT], stdout=targetfile, stdin=PIPE)
p.communicate(content)

sourcefile.close()
targetfile.close()

The data in sourcefile is quite large, so it takes a lot of memory/swap to store it in 'content'. I tried to send the file directly to stdin with stdin=sourcefile, which works except the external script 'hangs', ie: keeps waiting for an EOF. This might be a bug in the external script, but that is out of my control for now..

Any advice on how to send the large file to my external script?

Answer 1

Replace the p.communicate(content) with a loop which reads from the sourcefile, and writes to p.stdin in blocks. When sourcefile is EOF, make sure to close p.stdin.

sourcefile = open(filein, "r")
targetfile = open(pathout, "w")

p = Popen([SCRIPT], stdout=targetfile, stdin=PIPE)
while True:
    data = sourcefile.read(1024)
    if len(data) == 0:
        break
    p.stdin.write(data)
sourcefile.close()
p.stdin.close()

p.wait()
targetfile.close()