how to write a Python debugger/editor

Question

Sorry for the kind of general question. More details about what I want:

I want the user to be able to write some Python code and execute it. Once there is an exception which is not handled, I want the debugger to pause the execution, show information about the current state/environment/stack/exception and make it possible to edit the code.

I only want to have the special code block editable where the exception occurred and nothing else (for now). I.e. if it occurred inside a for loop, I only want to have the code block inside the for loop editable. It should be the latest/most recent code block which is in the user editor scope (and not inside some other libs or the Python libs). Under this conditions, it is always clear what code block to edit.

---

I already tried to investigate a bit how to do this, though I feel a bit lost.

The Python traceback doesn't give me directly the code block, just the function and the code line. I could calculate that back but that seems a bit hacky to me. Better (and more natural) would be if I could somehow get a reference to the code block in the AST of the code.

To get the AST (and operate on it, i.e. manipulate/edit), I probably will use the compiler (which is deprecated?) and/or the parser module. Or the ast module. Not sure though how I could recompile special nodes / code blocks in the AST. Or if I only can recompile whole functions.

Answer 1

Playing around with ast and compile (built-in) it seems that you could possibly use the NodeTransformer to modify some nodes... You can also edit them manually if you know what you're looking for.

test.py

print 'Dumb Guy'
x = 4 + 4
print x * 3

change.py

import ast
with open('test.py') as f:
    expr = f.read()

e = ast.parse(expr)
e.body[0].values[0].s = 'Cool Guy'       # Replace the string
e.body[1].targets[0].id = 'herring'      # Change x to herring
e.body[2].values[0].left.id = 'herring'  # Change reference to x to reference to herring
c = compile(e, '<string>', 'exec')
exec(c)

Ouput of change.py:

Cool Guy
24

You can also add code to the body this way (or replace elements in the usual way of replacing list elements):

p = ast.parse('print "Sweet!"', mode='single')
e.body.extend(p)

and then just recompile and exec:

c = compile(e, '<string>', 'exec')
exec(c)

You can replace function definitions or single lines that way. A function definition will have its own body, so if you added some function (or loop) you could access it with

e.body[N].body  # Replace N with the index of the FunctionDef object

However, the only way that I know of to execute a single ast object (_ast.Print or _ast.Assign or whatever) is to do something like this:

e2 = ast.parse('', mode='exec')
e2.body.append(e.body[0])
exec(compile(e2, '<string>', 'exec'))

which seems a bit hackish to me. As far as lines go - each object in the AST has a lineno attribute, so if you can retrieve the line number from the exception you can fairly easily figure out which statement threw the exception.

Of course this doesn't really solve the problem of rewinding the stack to the pre-exception state, which is what you really want to do, it sounds like. However, it might be possible to do such a thing via pdb.

Answer 2

I wonder if the HAP Remote Debugger for Python might be of any use to you? I don't think that they have live editing, but some of the debugging aspects might be useful nonetheless.

Answer 3

From what I have figured out in the meantime, this is not possible. At least not block-wise. It is possible to recompile the code per function but not per code-block because Python generates the code objects per function.

So the only way is to write an own Python compiler.