After a customer uploads an image I want to re-size an image to 8 different sizes and then send them up to S3 and save them to the file system. Obviously this could take a long time (seconds) and I would want to do it in a "multithreaded" way (I quote that because I don't care if it's actually multithreaded or if it's some other functionality). My understanding is that because of the GIL in Python you cannot successfully use multithreading for things like this because it uses the same process and therefore works exactly the same as if you did it in your code's normal flow (vs multithreading). Is there some other, better way of opening a new process in Python for this sort of task?
+2
A:
If you want to call an external process, use the subprocess module.
If on the other hand you want to side-step the GIL and spin of some Python task, use the multiprocessing module. It provides an interface very much like the threading package's but utilizes subprocesses so you are not bound by the constraints of the GIL.
jkp
2010-01-20 19:24:35
What would be the reason for using subprocess (ie, what would be an external process)?
orokusaki
2010-01-20 19:25:26
@orokusaki: say you want to process some file with some external utility, EG, you are processing video and you want ffmpeg to transcode it for you: in this case you'd use the subprocess module to fire up ffmpeg, pipe your input data to it and read it's output to get the processed data. If what you want is to run some long-running Python routine without being affected by the GIL however, the multiprocessing package is what you want.
jkp
2010-01-20 19:28:25