Making Python’s __init__ method magical

January 3, 2014 . By Reuven

In some programming languages, the idea of “reflection” is somewhat exotic, and takes a while to learn. In Python (and Ruby, for that matter), the language is both dynamic and open, allowing you to poke around in the internals of just about any object you might like.  Reflection is a natural part of working with these languages, which lends itself to numerous things that you might want to do.

Reflection in Python is easy because everything is an object, and every Python object has attributes, which we can list using “dir”.  For example, I can list the attributes of a string:

>>> s = 'abc'
>>> dir(s)
['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getslice__', '__gt__', '__hash__', '__init__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_formatter_field_name_split', '_formatter_parser', 'capitalize', 'center', 'count', 'decode', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'index', 'isalnum', 'isalpha', 'isdigit', 'islower', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']

Since everything is an object, including built-in classes, I can get the attributes of a base type, as well.  For example, I can get the attributes associated with the “str” class:

>>> dir(str)
['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getslice__', '__gt__', '__hash__', '__init__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_formatter_field_name_split', '_formatter_parser', 'capitalize', 'center', 'count', 'decode', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'index', 'isalnum', 'isalpha', 'isdigit', 'islower', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']

(If you see a great deal of overlap here between the string instance and the str type, that’s because of the way in which Python handles attribute scoping.  If you cannot find an attribute on an object, then Python looks for the attribute on the object’s class.)

Functions are also objects in Python, which means that we can list their attributes, as well:

def hello(name):
    print "Hello, %s" % name

>>> hello("Dolly")
Hello, Dolly
>>> dir(hello)
['__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__doc__', '__format__', '__get__', '__getattribute__', '__globals__', '__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_name']

Given an object and the name of one of its attributes, I can retrieve, set, or check for the existence of the attribute value with the builtin “getattr”, “setattr”, and “hasattr” functions.  For example, I can first check to see if an attribute has been defined on an object:

>>> hasattr(hello, 'x')
False

Then I can set the attribute:

>>> setattr(hello, 'x', 5)
>>> hello.x
5

Notice that this does indeed mean that I have added a new “x” attribute to my function “hello”. It might sound crazy that Python lets me set an attribute on a function, and even crazier that the attribute can be a random name.  But that’s a fact of life in Python; in almost every case, you can add or change just about any attribute on just about any object. Of course, if you do this to classes that you didn’t create, or that aren’t expecting you to change things, then you can end up causing all sorts of strange behavior.

To retrieve our attribute value, we use the builtin “getattr” function:

>>> getattr(hello, 'x')
5

Note that there isn’t any difference between saying “hello.x” and invoking getattr.  By the same token, there’s no difference between putting “hello.x” on the left side of an assignment, and using the builtin “setattr” function.

One of the places where we tend to set lots of attributes is in our __init__ methods. While many people think of __init__ as a constructor, that’s not really the case.  Rather, __init__ is invoked on our new object after it has been created; its job is to take the new, blank-slate object and add whatever attributes we’ll want and need during the rest of its life.  It’s thus common for us to do something like this:

class Foo(object):
    def __init__(self, x, y):
        self.x = x
        self.y = y

>>> f = Foo(10, 20)
>>> f.x
10
>>> f.y
20

The above is all pretty standard stuff, but let’s still walk through it: We create an instance of Foo.  This means that Python first creates a naked Foo object (using the __new__ method behind the scenes), which is then passes to Foo.__init__, along with the parameter values from our invocation.  Inside of the __init__ method, there are three local variables: self, which points to the new object that we just created, x, and y.  There are actually two x’s and two y’s in this method: The local variables x and y, and the attributes x and y that sit on self.  From Python’s perspective, there is no connection between the local x and self.x.  At the same time, we see an obvious, semantic connection between the two — which is precisely as it should be.

I was teaching a Python class this week, when someone complained to me about the fact that Python code has this repeated boilerplate code in each object’s __init__ method.  Why, he asked me, does Python not have a way to automatically take all of the local variables, and pass them directly as attributes on the instance?  That is, if I pass parameters ‘x’ and ‘y’ to my object instantiation, then the method shouldn’t have to be told to run self.x = x, or self.y = y.  Rather, Python could just do that automatically

This is almost certainly not something that we want to do in Python.  The language (and its community) loves to make things explicit rather than implicit (as stated in the Zen of Python), and this sort of black magic seems more appropriate for Ruby than Python.  But it did seem like an interesting challenge to find out how easily I could implement such a thing, and I gladly took the challenge.

In order to solve this problem, I decided to work backwards: My ultimate goal would be to set attributes on self. That is, I would want to invoke setattr on self, passing it the name of the attribute I want to set, plus the associated value.  Thus, if the function sees that it has parameters self, x, and y, it should invoke setattr(self, ‘x’, VALUE) and setattr(self, ‘y’, VALUE).  I say VALUE here, because we still haven’t figured out where we’ll get the attribute names from, let alone their values.

It’s actually not that hard, as a general rule, to get the attribute names from a function.  I can just go to any function and ask for the __code__ attribute. Underneath that, among other things, is a co_varnames attribute, containing the names of local variables defined in the function.  So if I can just get __code__.co_varnames from inside of __init__ when it is invoked, we’ll be in great shape. (Note that in Python 3, the names of these attributes have changed slightly.)

Well, sort of: It’s nice that __code__ is available as an attribute on the function, but those attributes are only available via the function’s name.  How can I refer to the function from within the function itself?  Is there a pointer to “the current function”?

Not directly, no.  But the “inspect” module, which comes with Python, is perfect for such cases.  Normally, we think of inspect as allowing us to look at Python objects from the outside.  But inspect also allows us to look at the current stack frame, which includes the function that is currently executing.  For example:

>>>frame = inspect.currentframe()
>>> frame.f_code
<code object <module> at 0x106a24830, file "<stdin>", line 1>

The above was executed outside of a function, which means that the function-related information is missing.  Things get much more interesting when we’re inside of a function, however: f_code returns the function object on which we’re working (i.e., the stuff that is usually under the function’s __code__ attribute):

def foo():
    print(inspect.currentframe().f_code.co_name)

>>> foo()
foo

We can also get other function attributes, such as the names of local variables:

def foo(x):
    print(inspect.currentframe().f_code.co_varnames)

>>> foo(5)
('x',)

As you can see, we can get the names of local variables — including parameter names — from the co_varnames attribute. A very simple version of our “autoinit” function could thus take an object and one or more parameters, the names of which would be used to set attributes on that object. For example:

def autoinit(obj, x, y):
    for varname in inspect.currentframe().f_code.co_varnames:
        if varname == 'obj':
            continue
        else:
            setattr(obj, varname, 'hello')

>>> import os
>>> autoinit(os, 5, 10)
>>> os.x
'hello'
>>> os.y
'hello'

In the above example, we define our function such that it’ll take the names of all local variables (except for “obj”) and assign new attributes to that object.   However, the value is always ‘hello’.  How can we assign the actual values that are being passed to the parameters?

The easiest solution, I think, is to use the locals() function, which returns a dictionary of the currently defined local variables.  The keys of this dictionary are strings, which means that we can pretty trivially use the variable names to grab the value — and then make the assignment:

def autoinit(obj, x, y):
    for varname in inspect.currentframe().f_code.co_varnames:
        if varname == 'obj':
            continue
        else:
            setattr(obj, varname, locals()[varname])

>>> autoinit(os, 5, 10)
>>> os.x
5
>>> os.y
10

So far, so good!  We now have a function that can use the parameter names in setting attributes.  However, if this function is going to work, then it’s not going to exist on its own. Rather, we want to invoke “autoinit” from within our __init__ method.  This means that we need autoinit to set attributes not based on its own parameters, but rather based on its caller’s parameters.  The inspect.currentframe method returns the current stack frame, but we want the caller’s stack frame.

Fortunately, the implementers of inspect.currentframe thought of this, and provide an easy and elegant solution: If we invoke inspect.currentframe with a numeric parameter, Python will jump back that number of stack frames.  Thus, inspect.currentframe() returns the current stack frame, inspect.currentframe(1) returns the caller’s stack frame, and inspect.currentframe(2) inspect the caller’s caller’s stack frame.

By invoking inspect.currentframe(1) from within __init__, we can get the instance onto which we want to add the attributes, as well as the attribute names and values themselves. For example:

import inspect
def autoinit():
    frame = inspect.currentframe(1)
    params = frame.f_locals
    self = params['self']
    paramnames = frame.f_code.co_varnames[1:] # ignore self
    for name in paramnames:
        setattr(self, name, params[name])

class Foo(object):
    def __init__(self, x, y):
        autoinit()

>>> f = Foo(100, 'abc')
>>> f.x
100
>>> f.y
'abc'
>>> g = Foo(200, 'ghi')
>>> g.x
200
>>> g.y
'ghi'

Hey, that’s looking pretty good!  However, there is still one problem: Python doesn’t see a difference between parameters and local variables.  This means that if we create a local variable within __init__, autoinit will get confused:

class Bar(object):
    def __init__(self, x, y):
        autoinit()
        z = 999
>>> b = Bar(100, 'xyz')
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "<stdin>", line 3, in __init__
 File "<stdin>", line 7, in autoinit
KeyError: 'z'

As you can see, autoinit tries to find the value of our ‘z’ local variable — but because the variable has not been set, it isn’t in the locals() dictionary.  But the bigger problem is that autoinit is trying to look for z at all — as a local variable, rather than a parameter, we don’t want it to be set as an attribute on self!

The solution is to use the co_argcount attribute to our code object, which says how many arguments our function takes. For example:

def foo(x):
    y = 100
    return inspect.currentframe()

>>> s = foo(5)
>>> print(s.f_code.co_argcount)
1
>>> print(s.f_code.co_varnames)
('x', 'y')

We can improve our implementation of autoinit by only taking the first co_argcount elements of co_varnames. So far as I can tell (and I don’t know if this is official, or just a convenient accident), the arguments always come first in co_varnames.  Our final version of autoinit thus looks like this:

def autoinit():
    frame = inspect.currentframe(1)
    params = frame.f_locals
    nparams = frame.f_code.co_argcount
    self = params['self']
    paramnames = frame.f_code.co_varnames[1:nparams]
    for name in paramnames:
        setattr(self, name, params[name]) 

Sure enough, if we try it:

class Foo(object):
    def __init__(self, x, y):
        autoinit()
        z = 100
>>> f = Foo(10, 20)
>>> print f.x
10
>>> print f.y
20

Success!  Of course, this implementation still doesn’t handle *args or **kwargs.  And as I wrote above, it’s very much not in the spirit of Python to have such magic happening behind the scenes.  Yet, I’ve found this to be an interesting way to discover and play with functions, the “inspect” module, and how arguments are handled.

If you liked this explanation, then you’ll likely also enjoy my ebook, “Practice Makes Python,” with 50 exercises meant to improve your Python fluency.

Related Posts

Level up your Python skills this August

Level up your Python skills this August

Prepare yourself for a better career, with my new Python learning memberships

Prepare yourself for a better career, with my new Python learning memberships

I’m banned for life from advertising on Meta. Because I teach Python.

I’m banned for life from advertising on Meta. Because I teach Python.
  • Richard Prosser says:

    I was teaching a Python class this week, when someone complained to me about the fact that Python code has this repeated boilerplate code in each object’s __init__ method. Why, he asked me, does Python not have a way to automatically take all of the local variables, and pass them directly as attributes on the instance? That is, if I pass parameters ‘x’ and ‘y’ to my object instantiation, then the method shouldn’t have to be told to run self.x = x, or self.y = y. Rather, Python could just do that automatically

    This is almost certainly not something that we want to do in Python. The language (and its community) loves to make things explicit rather than implicit (as stated in the Zen of Python), and this sort of black magic seems more appropriate for Ruby than Python. But it did seem like an interesting challenge to find out how easily I could implement such a thing, and I gladly took the challenge.

    ————————————————————————————————————————————————–
    IMO, this is such a common requirement that it should be part of the language. “Explicit is better than Implicit” is fine of course but, taken to extremes, would mean writing machine code! Perhaps “Beautiful is better than Ugly” would be more appropriate here and it has greater precedence in the Zen of Python.

    Unfortunately even if a suitable PEP were proposed, I doubt that it would be accepted because of the possible side-effects that could be introduced to legacy programs. So the practical solution may be a metaclass (awkward) or a base class and a corresponding ‘super’ call to set the local parameter copies; that class could then be part of the standard library.

  • Very nice example of introspection, thanks !

  • Matthew Fernandez says:

    One can do something similar with the following, albeit without introspection of your __init__ code:

    class Foo(collections.namedtuple('Bar', ['x', 'y'])):
    ...

  • Thanks! will look into it (and the aggressive yet interesting comments above :))

  • Terence Honles says:

    OMG the HORROR! Why are you doing this!? There’s two better ways! Keyword arguments or locals()! PLEASE don’t be spreading this information!.


    class AutoInited:
    def __init__(self, **kargs):
    # yes you can wrap this loop in `autoinit(self, **kargs)` if you want
    for key in kargs:
    setattr(self, key, kargs[key])

    class AutoInited2:
    def __init__(self, a, b):
    # same `autoinit(self, **locals())` as above or
    for key, value in locals().items():
    setattr(self, key, value)

    # alternate "autoinit" method
    def autoinit(self, **kargs)
    for key in kargs:
    setattr(self, key, kargs[key])

    If you wanted to really hide the magic you could create a base class which does this for you, but PLEASE don’t do crazy meta programming when Python has exactly what you need in a much simpler form.

    I was HONESTLY hoping you’d say you could also do even SOMETHING like this

    • Thanks for the comment; I thought that it was an interesting exercise, but not (as I indicate repeatedly) the sort of thing that I want people to actually put into production.

      I didn’t use locals() because I wanted to distinguish between parameters and plain-ol’ local variables. And yes, I could have inherited from another class, or used a decorator (which would be even more appropriate, I believe), but that wasn’t the point of this little intellectual exercise.

      • Your post remind me about Django ORM and SQL Alchemy where such a sugar is used to construct models. You can check these sources.

        In addition, our team was implemented the same and the developers were happy about such “autoinitializing”. We used locals() and engaged inheritance. And I have to say we don’t like passing to *args and **kwargs. Explicit arguments declarations are the must, that helps a lot in future development and code navigation and documentation and so on. Thus locals() was all what we need. ‘inspect’ is much ‘heavyer’ and unclear.

        > I didn’t use locals() because I wanted to distinguish between parameters and plain-ol’ local variables.
        To my mid args = [arg for arg in locals() if arg != self] at the very beginning of a method shouldn’t mess anything with have just passed args and kwargs.

  • I’ve been using the following one liner in the init function to accomplish the same thing

    class A(object):
    def __init__(self, x, y):
    self.__dict__.update((k,v) for k,v in locals().items() if k != ‘self’)

  • By accepting keyword args only you get good enough functionality while keeping it simple


    class Foo(object):
    ....def __init__(self, **kwargs):
    ........for (k,v) in kwargs.iteritems():
    ............setattr(self, k, v)

  • How about the following? Inherit from AutoSetter to get self-initialising __init__ arguments:

    class AutoSetter(object):
    def __init__(self, a, b, c):
    map(lambda (n, v): n is not ‘self’ and setattr(self, n, v), locals().items())

  • Here is a version implemented as a decorator.
    Actually I’d like NOT to use inspect, but I didn’t find any nice way to get the kwargs default values without it, any suggestion?

    —-

    import inspect

    def autoinit(decorated_init):
    def _wrap(*args, **kwargs):
    obj, nargs= args[0], args[1:]
    names = decorated_init.func_code.co_varnames[1:len(nargs)+1]
    nargs = {k: nargs[names.index(k)] for k in names}
    kwa_keys = decorated_init.func_code.co_varnames[len(nargs)+1:]
    kwa_defaults = inspect.getargspec(decorated_init).defaults
    for k in kwa_keys:
    nargs[k] = kwargs[k] if (k in kwargs) else kwa_defaults[kwa_keys.index(k)]
    for k, v in nargs.iteritems():
    setattr(obj, k, v)
    return decorated_init(*args, **kwargs)
    return _wrap

    Examples:

    class Foo():
    @autoinit
    def __init__(self, a, b, x=0, y=0):
    self.p = ‘Hello, I am set inside the __init__’
    self.s = self.a + self.b + self.x + self.y

    f = Foo(1, 2, x=120, y=5)
    for param in ‘abxyps’:
    print param, ‘=>’, getattr(f, param)
    f = Foo(31, 33, x=64)
    for param in ‘abxyps’:
    print param, ‘=>’, getattr(f, param)
    f = Foo(7, 121)
    for param in ‘abxyps’:
    print param, ‘=>’, getattr(f, param)

  • Something like this? Not cleaned up.

    class A(object):
    def __init__(self, x, y, z, **args):
    d = locals().copy()
    d.update(args)
    del d[‘args’]
    del d[‘self’]
    for k, v in d.iteritems():
    setattr(self, k, v)

  • pythonist says:

    I prefer to do the following:

    #tested with python 3
    class Foo:
    def __init__( self, a, b, c ):
    vars( self ).update( locals() )
    del self.self

    f = Foo( 1, 2, 3 )
    print( '%s %s %s' % ( f.c, f.b, f.a ) )

    Explanation: vars() provides you the underlying dictionary of an object. You can the use the update() Method of a dictionary with the current local variables which also includes the current parameters to a function. These variables are self, a, b and c. After the update you usually want to remove self.self.

      • Ian Epperson says:

        The following DOES work on 2.7, not sure about 3:

        #tested with python 2.7
        class Foo(object):
        def __init__( self, a, b, c ):
        self.__dict__.update( locals() )
        del self.self

        f = Foo( 1, 2, 3 )
        print( '%s %s %s' % ( f.c, f.b, f.a ) )

        The __dict__ provides the object instance dictionary.

  • I think you have a typo in a couple of code examples, the first two times you show the “autoinit” function, you define it as “autoinit” but then you call “foo” against os..

  • {"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}
    >