Programming Forums > Script Programming > Python

GoldenArms · Jul 5th, 2006, 6:59 PM

Can someone please explain metaclasses & metaprogramming? I read a few definitions but I dont understand why they are necessary.... thanks

Arevos · Jul 5th, 2006, 8:24 PM

Metaprogramming is the writing of programs that are designed to write/rewrite other programs (or often themselves). Python is relatively well equipped in this regard.

Metaclasses are to classes what classes are to objects. Or, to put it another way, if you instantiate a class, you get an object, whilst if you instantiate a metaclass, you get a class.

For instance, a string in Python is instantiated from the str object. The class of a string is str:

(Toggle Plain Text)

>>> print "Some string".__class__
<type 'str'>

Everything in Python is an object, including classes. Thus, if objects have classes, then classes have classes (or rather, metaclasses):

(Toggle Plain Text)

>>> "Some string".__class__.__class__
<type 'type'>

All new style classes in Python, such as str, inherit from the base class, object. In a similar way, all classes inherit from the metaclass, type.

Confusingly, type has two functions in Python. If type is called with one argument, it merely returns the type of an object. But if type is called with three arguments, it constructs a new class:

(Toggle Plain Text)

BlankClass = type("BlankClass", (), {})

The first argument is the class name. The second argument is a tuple of ancestor classes that this newly created class will derive from. The third argument is a dictionary that maps attributes to values and functions.

As well as using the generic type metaclass to construct a new class on the fly, one can dynamically alter existing classes by specifying a metaclass with the __metaclass__ attribute.

Before I touch on that, however, I'll give you a quick runthrough of decorators, and explain, roughly, what metaprogramming is.

Take this simple function:

(Toggle Plain Text)

def add(x, y):
	return x + y

If we wanted to add some debug logging to this function, we might put in code like so:

(Toggle Plain Text)

def add(x, y):
	print "Entering add"
	output = x + y
	print "Leaving add"
	return output

That's all well and good, and gives us some potentially useful output, but if we had to do this for every function, things would get a little boring. This is where decorators come in.

This is a decorator function:

(Toggle Plain Text)

def log(func):
	def logged(*args, **kwargs):
		print "Entering " + func.__name__
		output = func(*args, **kwargs)
		print "Leaving " + func.__name__
		return output
	return logged

Note that I have one function declared within another. The inner function, logged, is declared on the fly, each time the outer function, log, is executed. It's important to note that a new function is created each time log is executed. In other words, we're creating an entirely new function at runtime.

The classic way to use the decorator would be:

(Toggle Plain Text)

def add(x, y):
	return x + y
add = log(add)

But later versions of Python also provide a shortcut:

(Toggle Plain Text)

@log
def add(x, y):
	return x + y

The importance of this cannot be understated. This is a function that creates new functions. This is an example of higher level functions specifically, and of metaprogramming in general. In essence, we're creating programs that rewrite themselves as the situation dictates.

Moving back onto metaclasses. Let's say we had a class that looked something like:

(Toggle Plain Text)

class Foobar(object):
	@log
	def foo(self):
		print "foo"

	@log
	def bar(self):
		print "bar"

Each of the methods of the class uses the log decorator to log output. But if the class has many methods, typing @log out for every single method is tedidious and repetitive. Wouldn't it be nice if we could tell Python to add the log decorator to every class automatically? With metaclasses, we can!

(Toggle Plain Text)

from types import FunctionType

class Log(type):
	def __new__(cls, name, bases, members):
		functions = [v for v in members.values() 
		             if type(v) is FunctionType]

		for function in functions:
			members[function.__name__] = log(function)
					
		return type.__new__(cls, name, bases, members)

This metaclass takes all the functions in a class, and adds the log decorator to them. We can apply it to our test class, like so:

(Toggle Plain Text)

class Test(object):
	__metaclass__ = Log

	def foo(self):
		print "foo"

	def bar(self):
		print "bar"

Instant method decoration! Both foo and bar now output logging information, courtesy of the Log metaclass.

Phew! Sorry for going on for so long, but this is one of my favourite programming subjects

GoldenArms · Jul 5th, 2006, 9:17 PM

wow thanks alot! I have a way better grasp on it now.... One thing though, is it absolutley necessary for the logged function to be nested within log? Can't whats the point of having it nested?

Arevos · Jul 6th, 2006, 4:51 AM

Quote:

Originally Posted by GoldenArms

wow thanks alot! I have a way better grasp on it now.... One thing though, is it absolutley necessary for the logged function to be nested within log? Can't whats the point of having it nested?

The inner function, logged, refers to func, which is an argument of an outer variable. So log(add) would return a logged function for the function add, and log(foobar) would return a logged function for the function foobar:

(Toggle Plain Text)

def log(func):
	def logged(*args, **kwargs):
		# note how func comes from the outer function
		print "Entering " + func.__name__
		output = func(*args, **kwargs)
		print "Leaving " + func.__name__
		return output
	return logged

Or, to put it another way, log(add) and log(foobar) return two different functions.

I can probably illustrate this better with a simpler example:

(Toggle Plain Text)

def add(x):
	def inc(y):
		return x + y
	return inc

This is an addition function, similar to the addition function used in Haskell. It doesn't directly return the sum of two numbers; instead, it returns a function that will increment a number by a certain amount.

For instance:

(Toggle Plain Text)

>>> increment_by_one = add(1)
>>> increment_by_one(2)
3
>>> increment_by_three = add(3)
>>> increment_by_three(2)
5
>>> add(1)(2)
3
>>> add(3)(2)
5

Note that increment_by_one and increment_by_three produce different output for the same number. This shows that they are different functions.

The log function is similar, but instead of taking a number (x) from the outer function, it takes a function (func).

Sane · Jul 14th, 2006, 12:23 PM

Wow. That was definitely a good explanation Arevos. I had tried reading from a tutorial somewhere on the internet earlier, and it didn't explain well or help me very much. This cleared up almost everything for me. You should consider posting this in the tutorial section, or add it to one of the python tutorial threads.

Arevos · Jul 14th, 2006, 2:31 PM

Thanks for the encouragement, Sane

Another nice trick that I found whilst browsing the SQLObject codebase, is that __metaclass__ can be inherited. This can make adding metaclasses to a class somewhat neater:

(Toggle Plain Text)

class LogMixin(object):
	__metaclass__ = Log

class Test(LogMixin):
	def foo(self):
		print "foo"

	def bar(self):
		print "bar"

You can also add a metaclass to all classes in the current module:

(Toggle Plain Text)

__metaclass__ = Log

class Test:
	def foo(self):
		print "foo"

	def bar(self):
		print "bar"

Sane · Jul 14th, 2006, 11:47 PM

Ahh, I like that inheritance method. It really helps bundle your code in to logical groups. And it will give you quick control over which functions contain the metaclass, and which do not.

The second method seems like poor form however. Because, if you were writing a program, and you knew that you would need 50 classes that use the metaclass, it sounds like a good idea right? But what if at the end you need an unrelated class to not have a metaclass? You might have screwed yourself over. Perhaps you could add "__metaclass__ = None", I'm not sure. Regardless of that, it doesn't sound like the best way to go.

Cerulean · Jul 15th, 2006, 8:03 AM

If you have 50 classes that need the metaclass I'd assume they'd all be related enough to warrant their own module. This other stray class would not belong in that module. Problem solved

Jul 5th, 2006, 6:59 PM	#1
GoldenArms Newbie Join Date: Jul 2006 Posts: 8 Rep Power: 0	Metaprogramming Can someone please explain metaclasses & metaprogramming? I read a few definitions but I dont understand why they are necessary.... thanks

Jul 5th, 2006, 8:24 PM	#2
Arevos Programming Guru Join Date: Aug 2005 Location: England Posts: 1,499 Rep Power: 5	Metaprogramming is the writing of programs that are designed to write/rewrite other programs (or often themselves). Python is relatively well equipped in this regard. Metaclasses are to classes what classes are to objects. Or, to put it another way, if you instantiate a class, you get an object, whilst if you instantiate a metaclass, you get a class. For instance, a string in Python is instantiated from the str object. The class of a string is str: (Toggle Plain Text) >>> print "Some string".__class__ <type 'str'> >>> print "Some string".__class__ <type 'str'> Everything in Python is an object, including classes. Thus, if objects have classes, then classes have classes (or rather, metaclasses): (Toggle Plain Text) >>> "Some string".__class__.__class__ <type 'type'> >>> "Some string".__class__.__class__ <type 'type'> All new style classes in Python, such as str, inherit from the base class, object. In a similar way, all classes inherit from the metaclass, type. Confusingly, type has two functions in Python. If type is called with one argument, it merely returns the type of an object. But if type is called with three arguments, it constructs a new class: (Toggle Plain Text) BlankClass = type("BlankClass", (), {}) BlankClass = type("BlankClass", (), {}) The first argument is the class name. The second argument is a tuple of ancestor classes that this newly created class will derive from. The third argument is a dictionary that maps attributes to values and functions. As well as using the generic type metaclass to construct a new class on the fly, one can dynamically alter existing classes by specifying a metaclass with the __metaclass__ attribute. Before I touch on that, however, I'll give you a quick runthrough of decorators, and explain, roughly, what metaprogramming is. Take this simple function: (Toggle Plain Text) def add(x, y): return x + y def add(x, y): return x + y If we wanted to add some debug logging to this function, we might put in code like so: (Toggle Plain Text) def add(x, y): print "Entering add" output = x + y print "Leaving add" return output def add(x, y): print "Entering add" output = x + y print "Leaving add" return output That's all well and good, and gives us some potentially useful output, but if we had to do this for every function, things would get a little boring. This is where decorators come in. This is a decorator function: (Toggle Plain Text) def log(func): def logged(args, kwargs): print "Entering " + func.__name__ output = func(args, *kwargs) print "Leaving " + func.__name__ return output return logged def log(func): def logged(args, *kwargs): print "Entering " + func.__name__ output = func(args, *kwargs) print "Leaving " + func.__name__ return output return logged Note that I have one function declared within another. The inner function, logged, is declared on the fly, each time the outer function, log, is executed. It's important to note that a new function is created each time log* is executed. In other words, we're creating an entirely new function at runtime. The classic way to use the decorator would be: (Toggle Plain Text) def add(x, y): return x + y add = log(add) def add(x, y): return x + y add = log(add) But later versions of Python also provide a shortcut: (Toggle Plain Text) @log def add(x, y): return x + y @log def add(x, y): return x + y The importance of this cannot be understated. This is a function that creates new functions. This is an example of higher level functions specifically, and of metaprogramming in general. In essence, we're creating programs that rewrite themselves as the situation dictates. Moving back onto metaclasses. Let's say we had a class that looked something like: (Toggle Plain Text) class Foobar(object): @log def foo(self): print "foo" @log def bar(self): print "bar" class Foobar(object): @log def foo(self): print "foo" @log def bar(self): print "bar" Each of the methods of the class uses the log decorator to log output. But if the class has many methods, typing @log out for every single method is tedidious and repetitive. Wouldn't it be nice if we could tell Python to add the log decorator to every class automatically? With metaclasses, we can! (Toggle Plain Text) from types import FunctionType class Log(type): def __new__(cls, name, bases, members): functions = [v for v in members.values() if type(v) is FunctionType] for function in functions: members[function.__name__] = log(function) return type.__new__(cls, name, bases, members) from types import FunctionType class Log(type): def __new__(cls, name, bases, members): functions = [v for v in members.values() if type(v) is FunctionType] for function in functions: members[function.__name__] = log(function) return type.__new__(cls, name, bases, members) This metaclass takes all the functions in a class, and adds the log decorator to them. We can apply it to our test class, like so: (Toggle Plain Text) class Test(object): __metaclass__ = Log def foo(self): print "foo" def bar(self): print "bar" class Test(object): __metaclass__ = Log def foo(self): print "foo" def bar(self): print "bar" Instant method decoration! Both foo and bar now output logging information, courtesy of the Log metaclass. Phew! Sorry for going on for so long, but this is one of my favourite programming subjects Last edited by Arevos; Jul 5th, 2006 at 8:37 PM.

Jul 14th, 2006, 2:31 PM	#6
Arevos Programming Guru Join Date: Aug 2005 Location: England Posts: 1,499 Rep Power: 5	Thanks for the encouragement, Sane Another nice trick that I found whilst browsing the SQLObject codebase, is that __metaclass__ can be inherited. This can make adding metaclasses to a class somewhat neater: (Toggle Plain Text) class LogMixin(object): __metaclass__ = Log class Test(LogMixin): def foo(self): print "foo" def bar(self): print "bar" class LogMixin(object): __metaclass__ = Log class Test(LogMixin): def foo(self): print "foo" def bar(self): print "bar" You can also add a metaclass to all classes in the current module: (Toggle Plain Text) __metaclass__ = Log class Test: def foo(self): print "foo" def bar(self): print "bar" __metaclass__ = Log class Test: def foo(self): print "foo" def bar(self): print "bar"

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Jul 5th, 2006, 9:17 PM	#3
GoldenArms Newbie Join Date: Jul 2006 Posts: 8 Rep Power: 0	wow thanks alot! I have a way better grasp on it now.... One thing though, is it absolutley necessary for the logged function to be nested within log? Can't whats the point of having it nested?

Jul 14th, 2006, 12:23 PM	#5
Sane Programming Guru Join Date: Apr 2005 Location: Waterloo, Ontario Posts: 1,835 Rep Power: 5	Wow. That was definitely a good explanation Arevos. I had tried reading from a tutorial somewhere on the internet earlier, and it didn't explain well or help me very much. This cleared up almost everything for me. You should consider posting this in the tutorial section, or add it to one of the python tutorial threads.

Jul 14th, 2006, 11:47 PM	#7
Sane Programming Guru Join Date: Apr 2005 Location: Waterloo, Ontario Posts: 1,835 Rep Power: 5	Ahh, I like that inheritance method. It really helps bundle your code in to logical groups. And it will give you quick control over which functions contain the metaclass, and which do not. The second method seems like poor form however. Because, if you were writing a program, and you knew that you would need 50 classes that use the metaclass, it sounds like a good idea right? But what if at the end you need an unrelated class to not have a metaclass? You might have screwed yourself over. Perhaps you could add "__metaclass__ = None", I'm not sure. Regardless of that, it doesn't sound like the best way to go.

Jul 15th, 2006, 8:03 AM	#8
Cerulean Professional Programmer Join Date: Apr 2005 Location: London, England Posts: 459 Rep Power: 4	If you have 50 classes that need the metaclass I'd assume they'd all be related enough to warrant their own module. This other stray class would not belong in that module. Problem solved

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)