How can I debug a problem calling Python's copy.deepcopy() against a custom type?

Question

In my code I'm trying to take copies of instances a class using copy.deepcopy. The problem is that under some circumstances it is erroring with the following error:

TypeError: 'object.__new__(NotImplementedType) is not safe, use NotImplementedType.__new__()'

After much digging I have found that I am able to reproduce the error using the following code:

import copy
copy.deepcopy(__builtins__)

The problem appears to be that at some point it is trying to copy the NotImplementedType builtin. The question is why is it doing this? I have not overridden __deepcopy__ in my class and it doesn't happen all the time. Does anyone have any tips for tracking down where the request to make a copy of this type comes from?

I've put some debugging code in the copy module itself to ensure that this is what's happening, but the point at which the problem occurs is so far down a recursive stack it's very hard to make much of what I'm seeing.

Answer 1

you could override the __deepcopy__ method: (python documentation)

In order for a class to define its own copy implementation, it can define special methods __copy__() and __deepcopy__(). The former is called to implement the shallow copy operation; no additional arguments are passed. The latter is called to implement the deep copy operation; it is passed one argument, the memo dictionary. If the __deepcopy__() implementation needs to make a deep copy of a component, it should call the deepcopy() function with the component as first argument and the memo dictionary as second argument.

Otherwise you could save the modules in a global list or something else.

Answer 2

In the end I did some digging in the copy source code and came up with the following solution:

from copy import deepcopy, _deepcopy_dispatch
from types import ModuleType

class MyType(object):

    def __init__(self):
        self.module = __builtins__

    def copy(self):
        ''' Patch the deepcopy dispatcher to pass modules back unchanged '''
        _deepcopy_dispatch[ModuleType] = lambda x, m: x
        result = deepcopy(self)
        del _deepcopy_dispatch[ModuleType]
        return result

MyType().copy()

I realise this uses a private API but I couldn't find another clean way of achieving the same thing. I did a quick search on the web and found that other people had used the same API without any bother. If it changes in the future I'll take the hit.

I'm also aware that this is not thread-safe (if a thread needed the old behaviour whilst I was doing a copy on another thread I'd be screwed) but again its not a problem for me right now.

Hope that helps someone else out at some point.

Answer 3

You can override the deepcopy behavior of the class that contains a pointer to a module, by using the pickle protocol, which is supported by the copy module, as is stated here. In particular, you can define __getstate__ and __setstate__ for that class. E.g.:

>>> class MyClass:
...     def __getstate__(self):
...         state = self.__dict__.copy()
...         del state['some_module']
...         return state
...     def __setstate__(self, state):
...         self.__dict__.update(state)
...         self.some_module = some_module