Subprocess.Popen hangs with interactive programs when called from inside Django

Question

Hi,

I have written a small Django App, that executes an interactive program based on user input and returns the output as the result. But for some reason, the subprocess hangs. On verification of the logs I found that a place where a '\n' has to be given as repsonse to a challenge, the response seems to have never been made. Interestingly, if I run the same code from outside of Django, i.e either from a pyhton module or from the interactive shell, subprocess works without a hitch. I am assuming some settings within the environment used by Django are the culprit here. Here are snippets of the code that I've written:

def runtests(test_name, selective=False, tests_file=''):
    if selective:
        run_cmd = ['runtest', '--runfromfile', tests_file, test_name]
    else:
        run_cmd = 'runtest %s' % (test_name)
    print 'Executing command .. '
    print run_cmd
    p = subprocess.Popen(run_cmd, shell=False, stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT)

    return p.stdout.read()

def result(request):
    test_name = request.GET['test_name']
    if not test_name:
        return render_to_response('webrun/execute.html', {'error_flag':True})

    in_file = os.path.abspath('webrun/log/%s_in.xml' % test_name)
    suites = dict([(field[len('suite_'):],value) 
                            for field,value in request.GET.items() 
                            if field.startswith('suite_')])
    if suites:
        _from_dict_to_xml(suites, in_file, test_name)
    output = runtests(test_name, bool(suites), in_file)

    return render_to_response('webrun/result.html', {'output':output})

I've tried replacing subprocess with the older os.system method. But even that hangs in the exact same place. Again, this runs too if I were execute same code out of Django. I am new to Python and Django. Appreciate any insights into this issue. Thanks a heap.

Answer 1

The subprocess documentation suggests that you "use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process."

Does it work if you use return p.communicate()[0] instead of return p.stdout.read()?

Answer 2

I've had the same issue doing this sort of thing to get the mercurial tip version:

import subprocess

lProcess = subprocess.Popen(["hg","tip"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
lOutput = lProcess.stdout.readline()
lTipRevision = lOutput[10:].strip()
lTipRevision = lTipRevision[:lTipRevision.find(":")].strip()
print "Repository tip version is %s" % lTipRevision

This works fine when running through apache (mod_wsgi), but causes blank pages in the development server.

I've had a wander around the bugs related to this and the ones I could find seems to be closed as duplicate, or closed worksforme.

• http://code.djangoproject.com/ticket/9286
• http://code.djangoproject.com/ticket/4953
• http://code.djangoproject.com/ticket/3712

My original post about this is available at http://groups.google.com/group/django-users/browse_thread/thread/147ffe75899fd6e?pli=1

Answer 3

I think your problem is the directory where the 'runtest' program is located. You view is going to fetch that module in the same directory where the view is. You can also specify 'cwd' argument in the Popen list of arguments if the module is anywhere else. I'm using Popen command in a view with Django dev server whithout any problems, so the dev server is not the cause of your problem.

Answer 4

I'm also struggling with this, running a command that should not require any user input. Has anyone seen a solution? Running on Apache/mod_wsgi under CentOS.