Programming Forums - Metaprogramming

Programming Forums (http://www.programmingforums.org/forumindex.php)

- Python (http://www.programmingforums.org/forum43.html)

- - Metaprogramming (http://www.programmingforums.org/showthread.php?t=10633)

Can someone please explain metaclasses & metaprogramming? I read a few definitions but I dont understand why they are necessary.... thanks

Metaprogramming is the writing of programs that are designed to write/rewrite other programs (or often themselves). Python is relatively well equipped in this regard.

Metaclasses are to classes what classes are to objects. Or, to put it another way, if you instantiate a class, you get an object, whilst if you instantiate a metaclass, you get a class.

For instance, a string in Python is instantiated from the str object. The class of a string is str:

>>> print "Some string".__class__

<type 'str'>

Everything in Python is an object, including classes. Thus, if objects have classes, then classes have classes (or rather, metaclasses):

>>> "Some string".__class__.__class__

<type 'type'>

All new style classes in Python, such as str, inherit from the base class, object. In a similar way, all classes inherit from the metaclass, type.

Confusingly, type has two functions in Python. If type is called with one argument, it merely returns the type of an object. But if type is called with three arguments, it constructs a new class:

BlankClass = type("BlankClass", (), {})

The first argument is the class name. The second argument is a tuple of ancestor classes that this newly created class will derive from. The third argument is a dictionary that maps attributes to values and functions.

As well as using the generic type metaclass to construct a new class on the fly, one can dynamically alter existing classes by specifying a metaclass with the __metaclass__ attribute.

Before I touch on that, however, I'll give you a quick runthrough of decorators, and explain, roughly, what metaprogramming is.

Take this simple function:

def add(x, y):

        return x + y

If we wanted to add some debug logging to this function, we might put in code like so:

def add(x, y):

        print "Entering add"

        output = x + y

        print "Leaving add"

        return output

That's all well and good, and gives us some potentially useful output, but if we had to do this for every function, things would get a little boring. This is where decorators come in.

This is a decorator function:

def log(func):

        def logged(*args, **kwargs):

                print "Entering " + func.__name__

                output = func(*args, **kwargs)

                print "Leaving " + func.__name__

                return output

        return logged

Note that I have one function declared within another. The inner function, logged, is declared on the fly, each time the outer function, log, is executed. It's important to note that a new function is created each time log is executed. In other words, we're creating an entirely new function at runtime.

The classic way to use the decorator would be:

def add(x, y):

        return x + y

add = log(add)

But later versions of Python also provide a shortcut:

@log

def add(x, y):

        return x + y

The importance of this cannot be understated. This is a function that creates new functions. This is an example of higher level functions specifically, and of metaprogramming in general. In essence, we're creating programs that rewrite themselves as the situation dictates.

Moving back onto metaclasses. Let's say we had a class that looked something like:

class Foobar(object):

        @log

        def foo(self):

                print "foo"



        @log

        def bar(self):

                print "bar"

Each of the methods of the class uses the log decorator to log output. But if the class has many methods, typing @log out for every single method is tedidious and repetitive. Wouldn't it be nice if we could tell Python to add the log decorator to every class automatically? With metaclasses, we can!

from types import FunctionType



class Log(type):

        def __new__(cls, name, bases, members):

                functions = [v for v in members.values() 

                             if type(v) is FunctionType]



                for function in functions:

                        members[function.__name__] = log(function)

                                        

                return type.__new__(cls, name, bases, members)

This metaclass takes all the functions in a class, and adds the log decorator to them. We can apply it to our test class, like so:

class Test(object):

        __metaclass__ = Log



        def foo(self):

                print "foo"



        def bar(self):

                print "bar"

Instant method decoration! Both foo and bar now output logging information, courtesy of the Log metaclass.

Phew! Sorry for going on for so long, but this is one of my favourite programming subjects ;)

wow thanks alot! I have a way better grasp on it now.... One thing though, is it absolutley necessary for the logged function to be nested within log? Can't whats the point of having it nested?

Quote:

Originally Posted by GoldenArms

wow thanks alot! I have a way better grasp on it now.... One thing though, is it absolutley necessary for the logged function to be nested within log? Can't whats the point of having it nested?

The inner function, logged, refers to func, which is an argument of an outer variable. So log(add) would return a logged function for the function add, and log(foobar) would return a logged function for the function foobar:

def log(func):

        def logged(*args, **kwargs):

                # note how func comes from the outer function

                print "Entering " + func.__name__

                output = func(*args, **kwargs)

                print "Leaving " + func.__name__

                return output

        return logged

Or, to put it another way, log(add) and log(foobar) return two different functions.

I can probably illustrate this better with a simpler example:

def add(x):

        def inc(y):

                return x + y

        return inc

This is an addition function, similar to the addition function used in Haskell. It doesn't directly return the sum of two numbers; instead, it returns a function that will increment a number by a certain amount.

For instance:

>>> increment_by_one = add(1)

>>> increment_by_one(2)

3

>>> increment_by_three = add(3)

>>> increment_by_three(2)

5

>>> add(1)(2)

3

>>> add(3)(2)

5

Note that increment_by_one and increment_by_three produce different output for the same number. This shows that they are different functions.

The log function is similar, but instead of taking a number (x) from the outer function, it takes a function (func).

Wow. That was definitely a good explanation Arevos. I had tried reading from a tutorial somewhere on the internet earlier, and it didn't explain well or help me very much. This cleared up almost everything for me. You should consider posting this in the tutorial section, or add it to one of the python tutorial threads. :)

Thanks for the encouragement, Sane :)

Another nice trick that I found whilst browsing the SQLObject codebase, is that __metaclass__ can be inherited. This can make adding metaclasses to a class somewhat neater:

class LogMixin(object):

        __metaclass__ = Log



class Test(LogMixin):

        def foo(self):

                print "foo"



        def bar(self):

                print "bar"

You can also add a metaclass to all classes in the current module:

__metaclass__ = Log



class Test:

        def foo(self):

                print "foo"



        def bar(self):

                print "bar"

Ahh, I like that inheritance method. It really helps bundle your code in to logical groups. And it will give you quick control over which functions contain the metaclass, and which do not.

The second method seems like poor form however. Because, if you were writing a program, and you knew that you would need 50 classes that use the metaclass, it sounds like a good idea right? But what if at the end you need an unrelated class to not have a metaclass? You might have screwed yourself over. Perhaps you could add "__metaclass__ = None", I'm not sure. Regardless of that, it doesn't sound like the best way to go.

If you have 50 classes that need the metaclass I'd assume they'd all be related enough to warrant their own module. This other stray class would not belong in that module. Problem solved