Decorating with Python

I recently decided to build a miniature framework to make one of my personal projects a bit easier. In the course of doing so, I decided to use decorators to solve one of my first problem. A decorator in Python is an object that takes a function and returns a function. A callable object can be used as a decorator, so a standard Python function works just fine. More detailed information on decorators in Python can be found in PEP 318.

def exampleDecorator(func):
    print(str(func))
    return func

This decorator prints the name of the function when it decorates the function. To invoke it, use @ symbol when declaring a function.

@exampleDecorator
def double(p):
    return 2*p

The function double returns twice whatever the parameter p is. But the declaration above, tells the interpreter to decorate it using exampleDecorator. It's important to realize that functions are decorated when they are declared, not when they are invoked. Putting the two together

def exampleDecorator(func):
   print(str(func))
   return func

@exampleDecorator
def double(p):
   return 2*p

print(double(5))

This gives an output of

<function double at 0x1b7ec40>
10

In this example, I declared the decorator then declared double as a decorated function. The interpreter defines double as it normally would. Then, exampleDecorator is called and passed double as its parameter. All the example decorator does is print a string about the actual argument and return it. When double is called, it functions exactly as normal. This is because the decorator didn't actually change anything about the function. If you want to do something everytime double is invoked, you need a slightly more complex decorator. Let's define a decorator that only allows a single argument to be passed to a function, and tries to change any argument to an integer

def acceptInt(func):
    def wrapper(arg):
        arg = int(arg)
        return func(arg)
    return wrapper

At first glance this looks strange, nesting function definitions this way. The function acceptInt takes a single function func as its argument. It then declares another function that takes just one argument, tries to cast it to an integer, and passes it to func. It also returns whatever func returns. Let's decorate the double function again

@acceptInt
def double(p):
    return 2*p

print(double(5))
print(double('6'))

This gives an ouput of

10
12

The first line of output is the same as in the original example. Rightfully so, as double is declared exactly the same. The second line of output is twice the value of six. But notice when double was called, the string literal '6' was passed to it. Normally, 2*'6' would result in the string '66'. But since double was decorated with acceptInt the wrapper function declared in it converts '6' to the integer 6. Then wrapper then calls double as usual and returns the value from it.

Let's recap what we've learned so far. Decorators in Python are any object that can take a function and return a function. Decorators are attached to function by placing it directly above the function declaration and prepending the name of the decorator with '@'. Whenever the interpreter gets to the function declaration, it interprets the function as normal and then passes it to the decorator as an argument. At this point, the decorator can do anything it wants with the function including replacing it completely. This is what makes decorators powerful.

A real example

Consider the following function and decorator.

import types

def fileOpener(func):
    def wrapper(file,*args,**kwargs):
        if type(file) == types.StringType:
            with open(file) as fin:
                return func(fin,*args,**kwargs)

        return func(file,*args,**kwargs)

    return wrapper

@fileOpener
def searchFile(file,searchFor):
    for line in file.xreadlines():
        if searchFor in line:
            return line

The function searchFile iterates over each line in a file and returns the first line that matches. The arguments are a file object and the string to search for. If no lines match, it returns None. It is decorated with fileOpener. Let's search for the string 'root' in /etc/passwd. This string should almost always be present in that file.

with open('/etc/passwd') as fin:
    print(searchFile(fin,'root'))

print(searchFile('/etc/passwd','root'))

Running this gives an output of

root:x:0:0:root:/root:/bin/bash
root:x:0:0:root:/root:/bin/bash

Calling searchFile twice gives the same output in both cases. But the difference is in the second case, a string literal is passed as the first argument. But when searchFile calls xreadlines on the object, it works fine. This is because the wrapper function declared within fileOpener checks the first argument passed to the function. If it is a string, it treats it as a file path and opens the file, passing it on to searchFile. If is not a string, searchFile is invoked without any changes. This powerful idiom allows us to attach behavior to functions without actually changing the definition of the function.

Going farther

Previously we defined the fileOpener decorator that allowed a function that normally accepted a file object to be passed a string that is a file path and work as expected. However, it has the limitation that the decorator always expects the first argument to be the string or file object. You'd need to customize the decorator for each case. You could define fileOpener0, fileOpener1, etc. but that is obviously not very effecient. Instead, you can pass the decorator arguments. Let's go back to the example of requiring an argument to be of a specific type as in the acceptInt decorator. But let's generalize it to any type and argument

import inspect
import types
class accept(object):
    def __init__(self,argName,requiredType):
        self.requiredType = requiredType
        self.argName = argName

    def __call__(self,func):
        args,varargs,keywords,defaults = inspect.getargspec(func)
        argIndex = args.index(self.argName)
        def wrapper(*args):
            if type(args[argIndex]) != self.requiredType:
                args = list(args)
                args[argIndex] = self.requiredType(args[argIndex])

            return func(*args)
        return wrapper

@accept('searchFor',types.StringType)
def searchFile(f,searchFor):
    for line in f.xreadlines():
        if searchFor in line:
            return line

with open('/etc/passwd') as fin:
    print(searchFile(fin,'0'))

with open('/etc/passwd') as fin:    
    print(searchFile(fin,0))

This gives an output of

root:x:0:0:root:/root:/bin/bash

root:x:0:0:root:/root:/bin/bash

The searchFile function is the same as before. But the accept decorator is new and is declared as a class. This is fine because __call__ is also declared, and Python just expects a decorator to be callable. For more on what is callable in Python take a look here. The decorator also takes two arguments: the name of the argument and type it is required to be. When searchFile in invoked the first time a string is passed and matches a line. In the second call, the literal 0 is passed which is an int. But the same line in the file is matched. This works because the accept decorator converts it to a string.

The structure of the accept decorator may look a little out of place. Python actually passes the decorator arguments to the decorator in the constructor of the object. After that, it calls the constructed object and passes it just one argument: the function to be wrapped. It is expected that the decorator returns the wrapped function from the __call__ method. That's probably a little confusing, so here is the same function searchFile declared without the @ decorator syntax.

def searchFile(f,searchFor):
    for line in f.xreadlines():
        if searchFor in line:
            return line
searchFile = accept('searchFor',types.StringType)(searchFile)

The decorator is constructed once per use, immediately called, and then discarded. All the transformations on the wrapped function have to happen at that time.

Getting back to the way the accept decorator works, the __call__ method declares the function wrapper just like the acceptInt decorator. The inspect module is used to get the position of the argument specified in the decorator's constructor in args. The wrapper function checks the specified argument and converts it if the type is non-conformant. It's necessary to convert args to a list if the argument needs to be cast. When called, args is always a tuple which is immutable.

Readers may notice this example will not work with functions called using keyword arguments. The example could be extended but that is not the goal.

Multiple decorations

It is worth mentioning here that the same function can be decorated more than once. Decorators are called in ascending order. This is slightly counter-intuitive because we're used to read source code in a start to finish manner. Here is a contrived example to demonstrate this.

def deco0(func):
    print 'A'
    return func

def deco1(func):
    print 'B'
    return func

def deco2(func):
    print 'C'
    return func

@deco0    
@deco1
@deco2
def double(x):
    return 2*x

This gives an output of

C
B
A

As you can see, deco2 is called first. While we're at it, it is worth mentioning you can decorate the same function with the same decorator more than once. A use case hasn't been identified for this, however.

def deco(func):
    print 'A'
    return func

@deco   
@deco
@deco
def double(x):
    return 2*x

This gives an output of

A
A
A

Decorating instance methods

Being an object-oriented language, it's not unusual to want to decorate an instance method of an object. This requires understanding another idiosyncrasy of Python's: descriptors. Descriptors are a way of controlling the binding behavior of an object. To learn more about descriptor, start here. To decorate an instance method, a non-data descriptor is needed. This type of descriptor is an object implementing the __get__ method. The interpreter invokes this method to bind an object to a specific instance.

import copy
import types
import sys

class fileWriter(object):
    def __init__(self,default):
        self.default = default
    def __call__(self,func):
        return FileWriter(func,self.default)

class FileWriter(object):
    def __init__(self,wrapped,default):
        self.default = default
        self.wrapped = wrapped

    def __get__(self,instance,clazz):
        ret = copy.copy(self)
        ret.instance = instance
        return ret

    def __call__(self,*args):
        if len(args) == 0:
            args = [self.default]
        f = args[0]
        if type(f) == types.StringType:
            with open(f,'w') as fout:
                return self.wrapped(self.instance,fout)
        return self.wrapped(self.instance, f)

class BankAccount(object):
    def __init__(self,startingBalance):
        self.balance = startingBalance
        self.transactions = []

    def applyTransaction(self,amount):
        self.balance += amount

        if amount < 0:
            self.transactions.append(('withdrawal',amount,self.balance))
        else:
            self.transactions.append(('deposit',amount,self.balance))

    @fileWriter('default')
    def writeLog(self,fout):
        for action,amount,bal in self.transactions:
            fout.write(action + ' ' + str(amount)  + ' ' + str(bal) + '\n')

myAccount = BankAccount(1.0)
myAccount.applyTransaction(100.0)
myAccount.applyTransaction(-6.38)
myAccount.applyTransaction(-45.0)
myAccount.applyTransaction(350.0)

myAccount.writeLog(sys.stdout)
myAccount.writeLog('tmp')
myAccount.writeLog()

This gets written to standard output. It also gets written to the files 'tmp' and 'default'.

deposit 100.0 101.0
withdrawal -6.38 94.62
withdrawal -45.0 49.62
deposit 350.0 399.62

Let's understand what is going on with fileWriter and FileWriter. fileWriter is a decorator that is used to decorate BankAccount.writeLog. It doesn't do much, but does construct an instance of FileWriter. It returns that instance, replacing the BankAccount.writeLog in the BankAccount class. This is easy to see in the interactive interpreter

>>> print BankAccount.writeLog
<__main__.FileWriter object at 0x28cfe60>

The writeLog member of BankAccount is actually an instance of FileWriter even though it was declared as a function. It should be obvious to the reader that fileWriter and FileWriter work closely together. The naming is strictly a choice of the writer and is not convention.

What exactly is FileWriter? It's a non data descriptor. The Python interpreter calls the __get__ method when it wants to bind it to a specific instance. It passes in the instance and the class to the method. The class is safely ignored here. The __get__ method returns a copy of the FileWriter with the instance member filled in. This allows each instance of BankAccount to have it's own copy of FileWriter. Whenever writeLog is called on the myAccount instance, it's actually a copy of the FileWriter instance that is called. So the FileWriter.__call__ method is called. At this point, what it is doing should be obvious so I won't re-hash it.

As you can see, decorating an instance method is two-step ordeal involving created a decorator and an associtaed non-data descriptor. If you don't do this, you can still decorate methods of a class, but they wind up getting treated as non-instance methods even if you intended to declare an instance method.