I recently decided to build a miniature framework to make one of my personal projects a bit easier. In the course of doing so, I decided to use decorators to solve one of my first problem. A decorator in Python is an object that takes a function and returns a function. A callable object can be used as a decorator, so a standard Python function works just fine. More detailed information on decorators in Python can be found in PEP 318.
def exampleDecorator(func): print(str(func)) return func
This decorator prints the name of the function when it decorates the function. To invoke it, use
@ symbol when declaring a function.
@exampleDecorator def double(p): return 2*p
double returns twice whatever the parameter
p is. But the declaration above, tells the interpreter to decorate it using
exampleDecorator. It's important to realize that functions are decorated when they are declared, not when they are invoked. Putting the two together
def exampleDecorator(func): print(str(func)) return func @exampleDecorator def double(p): return 2*p print(double(5))
This gives an output of
<function double at 0x1b7ec40> 10
In this example, I declared the decorator then declared
double as a decorated function. The interpreter defines double as it normally would. Then,
exampleDecorator is called and passed
double as its parameter. All the example decorator does is print a string about the actual argument and return it. When
double is called, it functions exactly as normal. This is because the decorator didn't actually change anything about the function. If you want to do something everytime
double is invoked, you need a slightly more complex decorator. Let's define a decorator that only allows a single argument to be passed to a function, and tries to change any argument to an integer
def acceptInt(func): def wrapper(arg): arg = int(arg) return func(arg) return wrapper
At first glance this looks strange, nesting function definitions this way. The function
acceptInt takes a single function
func as its argument. It then declares another function that takes just one argument, tries to cast it to an integer, and passes it to
func. It also returns whatever
func returns. Let's decorate the
double function again
@acceptInt def double(p): return 2*p print(double(5)) print(double('6'))
This gives an ouput of
The first line of output is the same as in the original example. Rightfully so, as
double is declared exactly the same. The second line of output is twice the value of six. But notice when
double was called, the string literal
'6' was passed to it. Normally,
2*'6' would result in the string
'66'. But since
double was decorated with
wrapper function declared in it converts
'6' to the integer
wrapper then calls
double as usual and returns the value from it.
Let's recap what we've learned so far. Decorators in Python are any object that can take a function and return a function. Decorators are attached to function by placing it directly above the function declaration and prepending the name of the decorator with '@'. Whenever the interpreter gets to the function declaration, it interprets the function as normal and then passes it to the decorator as an argument. At this point, the decorator can do anything it wants with the function including replacing it completely. This is what makes decorators powerful.
A real example
Consider the following function and decorator.
import types def fileOpener(func): def wrapper(file,*args,**kwargs): if type(file) == types.StringType: with open(file) as fin: return func(fin,*args,**kwargs) return func(file,*args,**kwargs) return wrapper @fileOpener def searchFile(file,searchFor): for line in file.xreadlines(): if searchFor in line: return line
searchFile iterates over each line in a file and returns the first line that matches. The arguments are a file object and the string to search for. If no lines match, it returns
None. It is decorated with
fileOpener. Let's search for the string
/etc/passwd. This string should almost always be present in that file.
with open('/etc/passwd') as fin: print(searchFile(fin,'root')) print(searchFile('/etc/passwd','root'))
Running this gives an output of
searchFile twice gives the same output in both cases. But the difference is in the second case, a string literal is passed as the first argument. But when searchFile calls
xreadlines on the object, it works fine. This is because the
wrapper function declared within
fileOpener checks the first argument passed to the function. If it is a string, it treats it as a file path and opens the file, passing it on to
searchFile. If is not a string,
searchFile is invoked without any changes. This powerful idiom allows us to attach behavior to functions without actually changing the definition of the function.
Previously we defined the
fileOpener decorator that allowed a function that normally accepted a file object to be passed a string that is a file path and work as expected. However, it has the limitation that the decorator always expects the first argument to be the string or file object. You'd need to customize the decorator for each case. You could define
fileOpener1, etc. but that is obviously not very effecient. Instead, you can pass the decorator arguments. Let's go back to the example of requiring an argument to be of a specific type as in the
acceptInt decorator. But let's generalize it to any type and argument
import inspect import types class accept(object): def __init__(self,argName,requiredType): self.requiredType = requiredType self.argName = argName def __call__(self,func): args,varargs,keywords,defaults = inspect.getargspec(func) argIndex = args.index(self.argName) def wrapper(*args): if type(args[argIndex]) != self.requiredType: args = list(args) args[argIndex] = self.requiredType(args[argIndex]) return func(*args) return wrapper @accept('searchFor',types.StringType) def searchFile(f,searchFor): for line in f.xreadlines(): if searchFor in line: return line with open('/etc/passwd') as fin: print(searchFile(fin,'0')) with open('/etc/passwd') as fin: print(searchFile(fin,0))
This gives an output of
searchFile function is the same as before. But the
accept decorator is new and is declared as a
class. This is fine because
__call__ is also declared, and Python just expects a decorator to be callable. For more on what is callable in Python take a look here. The decorator also takes two arguments: the name of the argument and type it is required to be. When
searchFile in invoked the first time a
string is passed and matches a line. In the second call, the literal
0 is passed which is an
int. But the same line in the file is matched. This works because the
accept decorator converts it to a string.
The structure of the
accept decorator may look a little out of place. Python actually passes the decorator arguments to the decorator in the constructor of the object. After that, it calls the constructed object and passes it just one argument: the function to be wrapped. It is expected that the decorator returns the wrapped function from the
__call__ method. That's probably a little confusing, so here is the same function
searchFile declared without the
@ decorator syntax.
def searchFile(f,searchFor): for line in f.xreadlines(): if searchFor in line: return line searchFile = accept('searchFor',types.StringType)(searchFile)
The decorator is constructed once per use, immediately called, and then discarded. All the transformations on the wrapped function have to happen at that time.
Getting back to the way the
accept decorator works, the
__call__ method declares the function
wrapper just like the
acceptInt decorator. The
inspect module is used to get the position of the argument specified in the decorator's constructor in
wrapper function checks the specified argument and converts it if the type is non-conformant. It's necessary to convert
args to a list if the argument needs to be cast. When called,
args is always a
tuple which is immutable.
Readers may notice this example will not work with functions called using keyword arguments. The example could be extended but that is not the goal.
It is worth mentioning here that the same function can be decorated more than once. Decorators are called in ascending order. This is slightly counter-intuitive because we're used to read source code in a start to finish manner. Here is a contrived example to demonstrate this.
def deco0(func): print 'A' return func def deco1(func): print 'B' return func def deco2(func): print 'C' return func @deco0 @deco1 @deco2 def double(x): return 2*x
This gives an output of
C B A
As you can see,
deco2 is called first. While we're at it, it is worth mentioning you can decorate the same function with the same decorator more than once. A use case hasn't been identified for this, however.
def deco(func): print 'A' return func @deco @deco @deco def double(x): return 2*x
This gives an output of
A A A
Decorating instance methods
Being an object-oriented language, it's not unusual to want to decorate an instance method of an object. This requires understanding another idiosyncrasy of Python's: descriptors.
Descriptors are a way of controlling the binding behavior of an object. To learn more about descriptor, start here. To decorate an instance method, a non-data descriptor is needed. This type of descriptor is an object implementing the
__get__ method. The interpreter invokes this method to bind an object to a specific instance.
import copy import types import sys class fileWriter(object): def __init__(self,default): self.default = default def __call__(self,func): return FileWriter(func,self.default) class FileWriter(object): def __init__(self,wrapped,default): self.default = default self.wrapped = wrapped def __get__(self,instance,clazz): ret = copy.copy(self) ret.instance = instance return ret def __call__(self,*args): if len(args) == 0: args = [self.default] f = args if type(f) == types.StringType: with open(f,'w') as fout: return self.wrapped(self.instance,fout) return self.wrapped(self.instance, f) class BankAccount(object): def __init__(self,startingBalance): self.balance = startingBalance self.transactions =  def applyTransaction(self,amount): self.balance += amount if amount < 0: self.transactions.append(('withdrawal',amount,self.balance)) else: self.transactions.append(('deposit',amount,self.balance)) @fileWriter('default') def writeLog(self,fout): for action,amount,bal in self.transactions: fout.write(action + ' ' + str(amount) + ' ' + str(bal) + '\n') myAccount = BankAccount(1.0) myAccount.applyTransaction(100.0) myAccount.applyTransaction(-6.38) myAccount.applyTransaction(-45.0) myAccount.applyTransaction(350.0) myAccount.writeLog(sys.stdout) myAccount.writeLog('tmp') myAccount.writeLog()
This gets written to standard output. It also gets written to the files 'tmp' and 'default'.
deposit 100.0 101.0 withdrawal -6.38 94.62 withdrawal -45.0 49.62 deposit 350.0 399.62
Let's understand what is going on with
fileWriter is a decorator that is used to decorate
BankAccount.writeLog. It doesn't do much, but does construct an instance of
FileWriter. It returns that instance, replacing the
BankAccount.writeLog in the
BankAccount class. This is easy to see in the interactive interpreter
>>> print BankAccount.writeLog <__main__.FileWriter object at 0x28cfe60>
writeLog member of
BankAccount is actually an instance of
FileWriter even though it was declared as a function. It should be obvious to the reader that
FileWriter work closely together. The naming is strictly a choice of the writer and is not convention.
What exactly is
FileWriter? It's a non data descriptor. The Python interpreter calls the
__get__ method when it wants to bind it to a specific instance. It passes in the instance and the class to the method. The class is safely ignored here. The
__get__ method returns a copy of the
FileWriter with the
instance member filled in. This allows each instance of
BankAccount to have it's own copy of
writeLog is called on the
myAccount instance, it's actually a copy of the
FileWriter instance that is called. So the
FileWriter.__call__ method is called. At this point, what it is doing should be obvious so I won't re-hash it.
As you can see, decorating an instance method is two-step ordeal involving created a decorator and an associtaed non-data descriptor. If you don't do this, you can still decorate methods of a class, but they wind up getting treated as non-instance methods even if you intended to declare an instance method.