Python: How do I dynamically alter methods of dict and list objects?

Question

Here is a mockup of what I want to do:

alist = [1,2,3,4,5]   # create a vanilla python list object
replacef (alist)      # replace __setitem__, extend,... with custom functions
alist[0]=2            # now the custom __setitem__ is called

This is for a DSL project where the syntax should be as close to normal python as possible, so subclassing list and making the user call something like alist = MyList(1,2,3,4,5) is not desirable. Also, because the DSL needs to coexist with other libraries, globally changing list and dict is not an option...

I have tried creating instancemethods and setting them directly on the object like alist.append = myFunc, but Python says those properties are read only. Changing the __class__ attribute seems to be not permitted as well.

Is what I am trying to do even possible in Python?

Update: here are some findings on what is possible with a subclass of object:

Given:

class y(object):
    def __init__(self,a,b,c):
 self.a = a
 self.b = b
 self.c = c
def f(self):
 print self
 print self.a
 print self.b
 print self.c

class x(object):
def __init__(self,a,b,c):
 self.a = a
 self.b = b
 self.c = c
def f(self):
 print "x.f()"

>>> objy = y(1,2,3)
>>> objx = x(4,5,6)
>>> objy.f()
  <__main__.y object at 0x02612650>
  1
  2
  3
>>> objx.f()
  x.f()

It is possible to change the class of objx on the fly like this:

>>> objx.__class__ = y
>>> objx.f()
<__main__.y object at 0x02612D90>
4
5
6

Also, one can dynamically add/change object methods on the fly, much like in javascript:

>>> def g(self,p):
       print self
       print p
>>> import new
>>> objy.g = new.instancemethod(g,objy,y)
>>> objy.g(42)
<__main__.y object at 0x02612650>
42

However, both approaches will fail with objects that are implemented in C like dict and list:

>>> def mypop(self):
        print "mypop"
        list.mypop(self)


>>> alist.pop = new.instancemethod(mypop,alist,list) 

AttributeError: 'list' object attribute 'pop' is read-only

>>> x = [1,2,3,4,5]
>>> x.__class__ = mylist

TypeError: __class__ assignment: only for heap types

The suggested approach to use something like alist = replacef(alist) would not work here, because replacef could be called from a __setattribute__ call like this:

alist = [1,2,3,4,5]
aDSLObject.items = alist #  __setattribute__ calls replacef on alist
alist[0] = ....

So while it indeed is possible to change object methods dynamically, it seems like it is not possible to alter the behaviour of objects implemented in C - or am I missing something?

Answer 1

Maybe you can do the "alist = MyList(1,2,3,4,5)" thing inside the replaceref function?

def replacef(l):
    return MyList(l) # Return instance of list subclass that has custom __setitem__

alist = [1,2,3,4,5]
alist = replaceref(alist)

This way user would still use the standard list syntax when defining a list but would get the new setitem after replaceref call.

Answer 2

"Explicit is better than implicit." You should just document that your users need to use MyList() instead of list(). You and your users will be happier in the long run.

Ruby allows what you want, by the way. With Ruby, you can open up global objects and just add new behaviors. I find this scary, and Ruby runs slower than Python, so I'm not actually urging you to switch to Ruby.

So, provide MyList(), and provide a way to take an existing list and wrap it in a MyList().

You could write MyList() to just take a bunch of arguments, and make a list very conveniently:

class MyList(object):
    def __init__(self, *args):
        self.lst = args
    # add other methods to MyList() here

then you could call this with MyList(1, 2, 3, 4) and the instance would have a member called lst with the value: [1, 2, 3, 4]

But now, if you want to make a MyList instance out of a list, how do you do that? If you do this:

new_list = [1, 3, 5]
x = MyList(new_list)

then you have just set x to: [[1, 3, 5]]

So I recommend you make MyList() just take a list argument and save it:

class MyList(object):
    def __init__(self, lst):
        self.lst = list(lst)
    # add other methods to MyList() here

I put in list(lst) to handle the case where the lst argument is an iterator or something. If you want to be able to pass in a subclassed list type without stripping away the special subclass and "demoting" it to an ordinary list, you could just store the argument, and maybe put in a test to see if it is a list: isinstance(lst, list)