Can't pickle loggers??

Question

Working on a project that requires that I am able to pickle the container object at any point, since we expect it to fail on external conditions quite frequently and be able to statefully pick up where we left off.

I'm using the python logging library quite extensively, and all of my classes start by setting up a logger like:

class foo:
   def __init__(self):
       self.logger = logging.getLogger("package.foo")

Since I'm pickling a container class, it has several layers of classes within it, each with their own logger instance.

Now, for some reason, these loggers are breaking Pickle. I'm getting the following error, which goes away if I delete self.logger from all the classes:

Can't pickle 'lock' object: <thread.lock object at ... >

So my question is whether or not there is some way to remove the lock objects from all the loggers without having to recurse through my whole object tree deleting loggers which I will have to recreate on unpickle.

Answer 1

You could create a class that wraps the logger and implements __getstate__ and __setstate__.

This is pasted from http://docs.python.org/library/pickle.html. The fh is handled in a way which may be similar to what you need.

#!/usr/local/bin/python

class TextReader:
    """Print and number lines in a text file."""
    def __init__(self, file):
        self.file = file
        self.fh = open(file)
        self.lineno = 0

    def readline(self):
        self.lineno = self.lineno + 1
        line = self.fh.readline()
        if not line:
            return None
        if line.endswith("\n"):
            line = line[:-1]
        return "%d: %s" % (self.lineno, line)

    def __getstate__(self):
        odict = self.__dict__.copy() # copy the dict since we change it
        del odict['fh']              # remove filehandle entry
        return odict

    def __setstate__(self, dict):
        fh = open(dict['file'])      # reopen file
        count = dict['lineno']       # read from file...
        while count:                 # until line count is restored
            fh.readline()
            count = count - 1
        self.__dict__.update(dict)   # update attributes
        self.fh = fh                 # save the file object

Answer 2

Found a very similar question here, with an answer that worked for me:

http://stackoverflow.com/questions/2999638/how-to-stop-attributes-from-being-pickled-in-python