What are your (concrete) use-cases for metaclasses in Python?

Question

I have a friend who likes to use metaclasses, and regularly offers them as a solution.

I am of the mind that you almost never need to use metaclasses. Why? because I figure if you are doing something like that to a class, you should probably be doing it to an object. And a small redesign/refactor is in order.

Being able to use metaclasses has caused a lot of people in a lot of places to use classes as some kind of second rate object, which just seems disastrous to me. Is programming to be replaced by meta-programming? The addition of class decorators has unfortunately made it even more acceptable.

So please, I am desperate to know your valid (concrete) use-cases for metaclasses in Python. Or to be enlightened as to why mutating classes is better than mutating objects, sometimes.

I will start:

Sometimes when using a third-party library it is useful to be able to mutate the class in a certain way.

(this is the only case I can think of, and it's not concrete)

Answer 1

You never absolutely need to use a metaclass, since you can always construct a class that does what you want using inheritance or aggregation of the class you want to modify.

That said, it can be very handy in Smalltalk and Ruby to be able to modify an existing class, but Python doesn't like to do that directly.

There's an excellent DeveloperWorks article on metaclassing in Python that might help. The Wikipedia article is also pretty good.

Answer 2

The only time I used metaclasses in Python was when writing a wrapper for the Flickr API.

My goal was to scrape flickr's api site and dynamically generate a complete class hierarchy to allow API access using Python objects:

# Both the photo type and the flickr.photos.search API method 
# are generated at "run-time"
for photo in flickr.photos.search(text=balloons):
    print photo.description

So in that example, because I generated the entire Python Flickr API from the website, I really don't know the class definitions at runtime. Being able to dynamically generate types was very useful.

Answer 3

The purpose of metaclasses isn't to replace the class/object distinction with metaclass/class - it's to change the behaviour of class definitions (and thus their instances) in some way. Effectively it's to alter the behaviour of the class statement in ways that may be more useful for your particular domain than the default. The things I have used them for are:

• Tracking subclasses, usually to register handlers. This is handy when using a plugin style setup, where you wish to register a handler for a particular thing simply by subclassing and setting up a few class attributes. eg. suppose you write a handler for various music formats, where each class implements appropriate methods (play / get tags etc) for its type. Adding a handler for a new type becomes:

  class Mp3File(MusicFile):
      extensions = ['.mp3']  # Register this type as a handler for mp3 files
      ...
      # Implementation of mp3 methods go here
  

  The metaclass then maintains a dictionary of {'.mp3' : MP3File, ... } etc, and constructs an object of the appropriate type when you request a handler through a factory function.

• Changing behaviour. You may want to attach a special meaning to certain attributes, resulting in altered behaviour when they are present. For example, you may want to look for methods with the name _get_foo and _set_foo and transparently convert them to properties. As a real-world example, here's a recipe I wrote to give more C-like struct definitions. The metaclass is used to convert the declared items into a struct format string, handling inheritance etc, and produce a class capable of dealing with it.

  For other real-world examples, take a look at various ORMs, like sqlalchemy's ORM or sqlobject. Again, the purpose is to interpret defintions (here SQL column definitions) with a particular meaning.

Answer 4

Metaclasses aren't replacing programming! They're just a trick which can automate or make more elegant some tasks. A good example of this is Pygments syntax highlighting library. It has a class called RegexLexer which lets the user define a set of lexing rules as regular expressions on a class. A metaclass is used to turn the definitions into a useful parser.

They're like salt; it's easy to use too much.

Answer 5

Metaclasses can be handy for construction of Domain Specific Languages in Python. Concrete examples are Django, SQLObject 's declarative syntax of database schemata.

A basic example from A Conservative Metaclass by Ian Bicking:

The metaclasses I've used have been primarily to support a sort of declarative style of programming. For instance, consider a validation schema:

class Registration(schema.Schema):
    first_name = validators.String(notEmpty=True)
    last_name = validators.String(notEmpty=True)
    mi = validators.MaxLength(1)
    class Numbers(foreach.ForEach):
        class Number(schema.Schema):
            type = validators.OneOf(['home', 'work'])
            phone_number = validators.PhoneNumber()

Some other techniques: Ingredients for Building a DSL in Python (pdf).

Edit (by Ali): An example of doing this using collections and instances is what I would prefer. The important fact is the instances, which give you more power, and eliminate reason to use metaclasses. Further worth noting that your example uses a mixture of classes and instances, which is surely an indication that you can't just do it all with metaclasses. And creates a truly non-uniform way of doing it.

number_validator = [
    v.OneOf('type', ['home', 'work']),
    v.PhoneNumber('phone_number'),
]

validators = [
    v.String('first_name', notEmpty=True),
    v.String('last_name', notEmpty=True),
    v.MaxLength('mi', 1),
    v.ForEach([number_validator,])
]

It's not perfect, but already there is almost zero magic, no need for metaclasses, and improved uniformity.

Answer 6

Let's start with Tim Peter's classic quote:

Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don't (the people who actually need them know with certainty that they need them, and don't need an explanation about why). Tim Peters (c.l.p post 2002-12-22)

Having said that, I have (periodically) run across true uses of metaclasses. The one that comes to mind is in Django where all of your models inherit from models.Model. models.Model, in turn, does some serious magic to wrap your DB models with Django's ORM goodness. That magic happens by way of metaclasses. It creates all manner of exception classes, manager classes, etc. etc.

See django/db/models/base.py, class ModelBase() for the beginning of the story.

Answer 7

The way I used metaclasses was to provide some attributes to classes. Take for example:

class NameClass(type):
    def __init__(cls, *args, **kwargs):
       type.__init__(cls, *args, **kwargs)
       cls.name = cls.__name__

will put the name attribute on every class that will have the metaclass set to point to NameClass.

Answer 8

I have a class that handles non-interactive plotting, as a frontend to Matplotlib. However, on occasion one wants to do interactive plotting. With only a couple functions I found that I was able to increment the figure count, call draw manually, etc, but I needed to do these before and after every plotting call. So to create both an interactive plotting wrapper and an offscreen plotting wrapper, I found it was more efficient to do this via metaclasses, wrapping the appropriate methods, than to do something like:

class PlottingInteractive:
    add_slice = wrap_pylab_newplot(add_slice)

This method doesn't keep up with API changes and so on, but one that iterates over the class attributes in __init__ before re-setting the class attributes is more efficient and keeps things up to date:

class _Interactify(type):
    def __init__(cls, name, bases, d):
        super(_Interactify, cls).__init__(name, bases, d)
        for base in bases:
            for attrname in dir(base):
                if attrname in d: continue # If overridden, don't reset
                attr = getattr(cls, attrname)
                if type(attr) == types.MethodType:
                    if attrname.startswith("add_"):
                        setattr(cls, attrname, wrap_pylab_newplot(attr))
                    elif attrname.startswith("set_"):
                        setattr(cls, attrname, wrap_pylab_show(attr))

Of course, there might be better ways to do this, but I've found this to be effective. Of course, this could also be done in __new__ or __init__, but this was the solution I found the most straightforward.

Answer 9

I was thinking the same thing just yesterday and completely agree. The complications in the code caused by attempts to make it more declarative generally make the codebase harder to maintain, harder to read and less pythonic in my opinion. It also normally requires a lot of copy.copy()ing (to maintain inheritance and to copy from class to instance) and means you have to look in many places to see whats going on (always looking from metaclass up) which goes against the python grain also. I have been picking through formencode and sqlalchemy code to see if such a declarative style was worth it and its clearly not. Such style should be left to descriptors (such as property and methods) and immutable data. Ruby has better support for such declarative styles and I am glad the core python language is not going down that route.

I can see their use for debugging, add a metaclass to all your base classes to get richer info. I also see their use only in (very) large projects to get rid of some boilerplate code (but at the loss of clarity). sqlalchemy for example does use them elsewhere, to add a particular custom method to all subclasses based on an attribute value in their class definition e.g a toy example

class test(baseclass_with_metaclass):
    method_maker_value = "hello"

could have a metaclass that generated a method in that class with special properties based on "hello" (say a method that added "hello" to the end of a string). It could be good for maintainability to make sure you did not have to write a method in every subclass you make instead all you have to define is method_maker_value.

The need for this is so rare though and only cuts down on a bit of typing that its not really worth considering unless you have a large enough codebase.