Piping output of subprocess.call to progress bar

Question

I'm using growisofs to burn an iso through my Python application. I have two classes in two different files; GUI() (main.py) and Boxblaze() (core.py). GUI() builds the window and handles all the events and stuff, and Boxblaze() has all the methods that GUI() calls.

Now when the user has selected the device to burn with, and the file to be burned, I need to call a method that calls the following command:`

growisofs -use-the-force-luke=dao -use-the-force-luke=break:1913760 -dvd-compat -speed=2 -Z /burner/device=/full/path/to.iso

This command should give an output similar to this:

Executing 'builtin_dd if=/home/nevon/games/Xbox 360 isos/The Godfather 2/alls-tgod2.iso of=/dev/scd0 obs=32k seek=0'
/dev/scd0: "Current Write Speed" is 2.5x1352KBps.
#more of the lines below, indicating progress.
7798128640/7835492352 (99.5%) @3.8x, remaining 0:06 RBU 100.0% UBU  99.8%
7815495680/7835492352 (99.7%) @3.8x, remaining 0:03 RBU  59.7% UBU  99.8%
7832862720/7835492352 (100.0%) @3.8x, remaining 0:00 RBU   7.9% UBU  99.8%
builtin_dd: 3825936*2KB out @ average 3.9x1352KBps
/dev/burner: flushing cache
/dev/burner: closing track
/dev/burner: closing disc

This command is run in a method called burn() in Boxblaze(). It looks simply like this:

def burn(self, file, device):
    subprocess.call(["growisofs", '-dry-run', "-use-the-force-luke=dao", "-use-the-force-luke=break:1913760", "-dvd-compat", "-speed=2", "-Z",  device +'='+ file])

Now my questions are the following:

1. How can I get the progress from the output (the percentage in brackets) and have my progress bar be set to "follow" that progress? My progress bar is called in the GUI() class, as such:

   get = builder.get_object

   self.progress_window = get("progressWindow")

   self.progressbar = get("progressbar")

2. Do I have to run this command in a separate thread in order for the GUI to remain responsive (so that I can update the progress bar and allow the user to cancel the burn if they want to)? If so, how can I do that and still be able to pass the progress to the progress bar?

---

The full code is available on Launchpad if you are interested. If you have bazaar installed, just run:

bzr branch lp:boxblaze

Oh, and in case you were wondering, this application is only meant to work in Linux - so don't worry about cross-platform compatibility.

Answer 1

To get the output you need to use the subprocess.Popen call. (stdout = subprocess.PIPE)

(Second Question)

You probably need a separate thread, unless the GUI framework can select on a filedescriptor in the normal loop.

You can have a background thread read the pipe, process it (to extract the progress), the pass that to the GUI thread.

## You might have to redirect stderr instead/as well
proc = sucprocess.Popen(command,stdout=subprocess.PIPE)
for line in proc.stdout.readlines():
    ## might not work - not sure about reading lines
    ## Parse the line to extract progress
    ## Pass progress to GUI thread

Edit:

I'm afraid I don't want to waste lots of CDs testing it out, so I haven't run it, but by you're comment it looks like it's not outputing the info to stdout, but to stderr.

I suggest running a sample command directly on the command-line, and redirecting stdout and stderr to different files.

growisofs [options] >stdout 2>stderr

Then you can work out which things come out on stdout and which on stderr.

If the stuff you want come on stderr, change stdout=subprocess.PIPE to stderr=subprocess.PIPE and see if that works any better.

Edit2:

You're not using threads correctly - you should be starting it - not running it directly.

Also:

gtk.gdk.threads_init()
threading.Thread.__init__(self)

is very weird - the initialiser calls should be in the initialiser - and I don't think you need to make it a gtk thread?

The way you call the run() method, is weird itself:

core.Burning.run(self.burning, self.filechooser.get_filename(), self.listofdevices[self.combobox.get_active()])

Call instance methods through the object:

self.burning.run(self.filechooser.get_filename(), self.listofdevices[self.combobox.get_active()])

(But you should have an __init__() method)

It seems to me that you are trying to run before you can walk. Try writing some simple threading code, then some simple code to run growisofs and parse the output, then some simple gtk+background threading code, and only then try combining them all together. In fact first start writing some simple Object oriented code, so that you understand methods and object first.

e.g. All classes you create in python should be new-style classes, you should call super-class initialisers from your initialiser etc.

Answer 2

Can you pass a timeout to reading from subprocess? I guess you can because my subProcess module was used as design input for it. You can use that to growiso.read(.1) and then parse and display the percentage from the outdata (or maybe errdata).

Answer 3

You need to run the command from a separate thread, and update the gui with gobject.idle_add calls. Right now you have a class "Burning" but you are using it wrong, it should be used like this:

self.burning = core.Burning(self.filechooser.get_filename(), self.listofdevices[self.combobox.get_active()], self.progressbar)
self.burning.start()

Obviously you will have to modify core.Burning. Then you will have access to the progressbar so you could make a function like this:

def set_progress_bar_fraction(self, fraction): 
    self.progress_bar.set_fraction(fraction)

Then every percentage update call it like this: gobject.idle_add(self.set_progress_bar_fraction, fraction)

More info on pygtk with threads here.

Answer 4

You can use glib.io_add_watch() to watch for output on the pipes connected to stdout and stderr in the subprocess object.

proc = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout_id = glib.io_add_watch(proc.stdout, glib.IO_IN|glib.IO_HUP, stdout_cb)
stderr_id = glib.io_add_watch(proc.stderr, glib.IO_IN|glib.IO_HUP, stderr_cb)

Then when the callback is called it should check for the condition and reads all the data from the pipe and processes it to get the info to update the ProgressBar. If the app buffers io then you may have to use a pty to fool it into thinking it's connected to a terminal so it will output a line at a time.