2

Sharpen your Python skills with Weekly Python Exercise

A new WPE cohort starts on July 2nd! Join now, and take advantage of early-bird pricing.

It’s time for another cohort of Weekly Python Exercise! This time, it’s an advanced cohort with 15 weeks of practice in such subjects as functional programming, object-oriented programming, iterators, generators, and decorators.

Early-bird pricing ends in just one week, on June 18th!

Learn more, and get a sample, at https://WeeklyPythonExercise.com/.

3

Why do Python lists let you += a tuple, when you can’t + a tuple?

Let’s say you have a list in Python:

>>> mylist = [10, 20, 30]

You want to add something to that list. The most standard way to do this is with the “append” method, which adds its argument to the end of the list:

>>> mylist.append(40)
>>> print(mylist)
[10, 20, 30, 40]

But what if you want to add multiple items to a list? If you’re new to Python, then you might think that you can and should use a “for” loop. For example:

>>> mylist = [10, 20, 30]
>>> new_items = [40, 50, 60]
>>> for one_item in new_items:
        mylist.append(one_item)
>>> print(mylist)
[10, 20, 30, 40, 50, 60]

Great, right? But it turns out that there is a smarter and faster way to do this. You can use the += operator. This operator, which invokes the “iadd” (“inplace add”) method on the object to its left, effectively does what we did above, but in much less code:

>>> mylist = [10, 20, 30]
>>> new_items = [40, 50, 60]
>>> mylist += new_items
>>> print(mylist)
[10, 20, 30, 40, 50, 60]

It’s not a huge surprise that += can do this. After all, we normally expect += to add and assign to the variable on its left; it works with numbers and strings, as well as other types. And we know that we can use the + operator on lists, too:

>>> [1, 2, 3] + [4, 5, 6]
[1, 2, 3, 4, 5, 6]

Can we join a list and a tuple? Let’s check:

>>> mylist = [10, 20, 30]
>>> t = (40, 50, 60)
>>> mylist + t
Traceback (most recent call last):
File "", line 1, in 
TypeError: can only concatenate list (not "tuple") to list

In other words: No. Trying to add a list and a tuple, even if we’re not affecting either, results in the above error.

Which is why it’s so surprising to many of my students that the following does work:

>>> mylist = [10, 20, 30]
>>> t = (40, 50, 60)
>>> mylist += t
>>> mylist
[10, 20, 30, 40, 50, 60]         

That’s right: Adding a list to a tuple with + doesn’t work. But if we use +=, it does.

What gives?

It’s common, when teaching Python, to say that

x += 5

is basically a rewrite of

x = x + 5

And in the majority of cases, that’s actually true. But it’s not always true.

Consider: When you say “x + y” in Python, the “+” operator is translated into a method call. Behind the scenes, no matter what “x” and “y” are, the expression is translated into:

x.__add__(y)

The “__add__” magic method is what’s invoked on an object when it is added to another object. The object on the right-hand side of the “+” is passed as an argument to the method, while the object on the left-hand side is the recipient of the method call. That’s why, if you want your own objects to handle the “+” operator, you need to define the “__add__” method in your class definition. Do that, and things work just fine.

And thus, when we say “x = x + 5”, this is turned into

x = x.__add__(5)

Meaning: First invoke the method, and then assign it back to the variable “x”. In this case, “x” isn’t changing; rather, the variable is now referencing a new object.

Now consider the “+=” operator: It’s translated by Python into “__iadd__”, short for “inplace add.” Notice the slightly different syntax that we use here:

x += y

is translated into

x.__iadd__(y)

Did you see the difference between __add__ and __iadd__? The latter executes the assignment all by itself, internally. You don’t have to capture its output and assign it back to x.

It turns out that the implementation of list.__iadd__ takes the second (right-hand side) argument and adds it, one element at a time, to the list. It does this internally, so that you don’t need to execute any assignment after. The second argument to “+=” must be iterable; if you say

mylist += 5

you will get an error, saying that integers are not iterable. But if you put a string, list, tuple, or any other iterable type on the right-hand side, “+=” will execute a “for” loop on that object, adding each of its elements, one at a time, to the list.

In other words: When you use + on a list, then the right-hand object must be a list. But when you use +=, then any iterable type is acceptable:

>>> mylist = [10, 20, 30]
>>> mylist += [40, 50]       # list
>>> mylist
[10, 20, 30, 40, 50]

>>> mylist += (60, 70)       # tuple
>>> mylist
[10, 20, 30, 40, 50, 60, 70]

>>> mylist += 'abc'          # string
>>> mylist
[10, 20, 30, 40, 50, 60, 70, 'a', 'b', 'c']

>>> mylist += {'x':1, 'y':2, 'z':3}    # dict!
>>> mylist
[10, 20, 30, 40, 50, 60, 70, 'a', 'b', 'c', 'x', 'y', 'z']

Does this work with other types? Not really. For example:

>>> t = (10, 20, 30)
>>> t += [40, 50]
Traceback (most recent call last):
File "", line 1, in 
TypeError: can only concatenate tuple (not "list") to tuple         

What happened here? Let’s check the definition of tuple.__iadd__ to find out:

>>> help(tuple.__iadd__)
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: type object 'tuple' has no attribute '__iadd__'

Wait a second: There is no “__iadd__” method for tuples? If so, then how can “+=” work at all?

Because Python tries to be smart in such cases: If the object implements “__iadd__”, then the “+=” operator invokes it. But if the object lacks an “__iadd__” implementation, then Python does what we all guess it normally does — namely, invoke “__add__”, and then assign the results back to the variable. For example:

>>> class Foo(object):
        def __init__(self, x):
            self.x = x
        def __add__(self, other):
            print("In __add__")
            return Foo(self.x + other.x)

>>> f1 = Foo(10)
>>> f2 = Foo(20)
>>> f1 += f2
In __add__
>>> vars(f1)
{'x': 30}         

In other words, Python notices that our Foo class lacks an implementation of “__iadd__”, and substitutes “__add__” for it, assigning its result (a new instance of Foo) to the original variable.

But if we add (so to speak) the right method, then it’s invoked:

>>> class Foo(object):
        def __init__(self, x):
            self.x = x
        def __add__(self, other):
            print("In __add__")
            return Foo(self.x + other.x)
        def __iadd__(self, other):
            print("In __iadd__")
            self.x = self.x + other.x
            return self         
>>> f1 = Foo(10)
>>> f2 = Foo(20)
>>> f1 += f2
In __iadd__
>>> vars(f1)
{'x': 30}         

In the case of Python lists, __iadd__ was implemented such that it doesn’t just add “other.x” to its own value, but that it iterates over each element of “other.x” in a “for” loop. And thus, while “__add__” with a tuple won’t work, “__iadd__” with just about every iterable data types will.

1

Registration (and early-bird pricing) is open for Weekly Python Exercise

Do you use Python, but sometimes feel stuck?

Do you visit Stack Overflow every time you want to solve a problem?

Do you wish that you understood how to use advanced techniques, such as generators and decorators, better?

If so, then good news: I’m opening a new cohort of Weekly Python Exercise, specifically aimed at intermediate/advanced developers! For 15 weeks, starting on July 2nd, you’ll be able to improve your Python fluency — just as many other Python developers from around the world have done over the last three years.

This cohort works the same as all the others in the WPE family: On Tuesday, you get a new question (along with “pytest” tests), posing a problem for you to solve. On the following Monday, you get the solution and a detailed explanation. In between, you can discuss the question with others in your cohort via our private forum.What topics will be considered in

Among the topics we’ll discuss in this cohort:

  • Iterators and generators
  • Decorators
  • Advanced object-oriented techniques
  • Advanced data structures
  • Functional programming techniques
  • Threads and processes

If you register by June 18th, then the price of this cohort is $80. It’ll then go up to $100 on June 19th, and then $120 in the final week before it starts. So sign up now for this cohort — and improve your Python fluency, and save some money along the way

Wondering what WPE is like? You can read more on at https://WeeklyPythonExercise.com/, as well as sign up for sample exercises.

Are you a student, pensioner/retiree/senior, or do you live outside of the world’s 30 richest countries? Then you’re entitled to a discount; just e-mail me at reuven@lerner.co.il for the appropriate coupon code.

Many hundreds of Python developers from around the world have leveled up their Python skills with Weekly Python Exercise. Join this course, use Stack Overflow less, and get more done at work — and maybe even a better job.

Python dicts and memory usage

Let’s say that we create a new, empty Python dictionary:

>>> d = {}

How much memory does this new, empty dict consume? We can find out with “sys.getsizeof“:

>>> import sys
>>> sys.getsizeof(d)
240

In other words, our dictionary, with nothing in it at all, consumes 240 bytes. Not bad; given how often dictionaries are used in Python, it’s good to know that they don’t normally consume that much memory.

What if I add something to the dict? What will happen to the memory usage?

>>> d['a'] = 1
>>> sys.getsizeof(d)
240

Something seems a bit fishy here, right? How can it be that our newly created dictionary, with zero key-value pairs, takes up the same space in memory as our dictionary with one key-value pair?

The answer is that “sys.getsizeof” is returning the size of the dictionary as a data structure, not the data inside of it. In other words: When we first create a dictionary, it contains eight slots that can be filled with key-value pairs. Only when the dictionary needs to grow, because it has too many key-value pairs for its current size, does it allocate more memory.

Moreover, the key-value pairs themselves aren’t stored in the dict itself. Rather, just a reference to the place in memory that holds the keys and values is stored there. So neither the type nor the size of the data is kept in the dictionary, and it certainly doesn’t affect the result of “sys.getsizeof” for the dictionary. Indeed, watch this:

>>> d['a'] = 'a' * 100000
>>> sys.getsizeof(d)
240

Even when the value is 100,000 characters long, our dictionary only needs 240 bytes.

What happens as we expand our dictionary? When does it request more memory? Let’s take a look:

>>> d = {}
>>> for one_letter in 'abcdefghijklmnopqrstuvwxyz':
d[one_letter] = one_letter
print(f'{len(d)}, sys.getsizeof(d) = {sys.getsizeof(d)}')

1, sys.getsizeof(d) = 240
2, sys.getsizeof(d) = 240
3, sys.getsizeof(d) = 240
4, sys.getsizeof(d) = 240
5, sys.getsizeof(d) = 240
6, sys.getsizeof(d) = 368
7, sys.getsizeof(d) = 368
8, sys.getsizeof(d) = 368
9, sys.getsizeof(d) = 368
10, sys.getsizeof(d) = 368
11, sys.getsizeof(d) = 648
12, sys.getsizeof(d) = 648
13, sys.getsizeof(d) = 648
14, sys.getsizeof(d) = 648
15, sys.getsizeof(d) = 648
16, sys.getsizeof(d) = 648
17, sys.getsizeof(d) = 648
18, sys.getsizeof(d) = 648
19, sys.getsizeof(d) = 648
20, sys.getsizeof(d) = 648
21, sys.getsizeof(d) = 648
22, sys.getsizeof(d) = 1184
23, sys.getsizeof(d) = 1184
24, sys.getsizeof(d) = 1184
25, sys.getsizeof(d) = 1184
26, sys.getsizeof(d) = 1184

As you can see, the dictionary adds more key-value pairs, it needs more memory. But it doesn’t grow with each addition; each time it needs more space, it allocates more than it needs, so that the allocations can be relative rare.

What happens if we remove items from our dictionary? Will it return memory to the system? Let’s find out:

>>> for key in list(d.keys()):
d.pop(key)

>>> len(d)

0

Notice that in the above code, I didn’t iterate over “d” or “d.keys”. Doing so would have led to an error, because changing a dictionary while iterating over it is a problem. I thus created a list based on the keys, and iterated over that.

You can also see that after removing these name-value pairs from my dict, it is indeed empty. And its memory usage?

>>> sys.getsizeof(d)
1184

In other words: Even though we’ve removed items from our dict, it hasn’t released the memory that it previously allocated. Of course, given how rarely I find myself removing items from dicts in actual Python code, I’m not hugely surprised that this happens. After all, why return memory to the system if you’re unlikely to need to do that? But it means that if you do allocate tons of memory to a dict, then you’re unlikely to get it back until the program ends, even if you remove items.

But wait: What if I remove everything from the dict? There’s a method, “dict.clear“, that does this. I don’t use it very often, but it might at least provide us with some useful data:

>>> d.clear()
>>> len(d)
0
>>> sys.getsizeof(d)
72

Wait a second here: After running “dict.clear”, our dict size is indeed 0. Which is what it was before. But we’re somehow using less memory than we even did at the start, when we created an empty dict! How can that be?

It would seem that when you run “dict.clear”, it removes not only all of the key-value pairs, but also that initial allocation of memory that is done for new, empty dictionaries. Meaning that we now have an “emptier than new” dictionary, taking up a paltry 72 bytes in our system.

If we add a new key-value pair to our dict, then if my theory is right, we should get back to the original size of 240 bytes:

>>> d['a'] = 1
>>> len(d)
0
>>> sys.getsizeof(d)
240

Sure enough, adding that one key-value pair to “d” forced the dictionary to allocate the same amount of memory it had before, back when we first created it.

Weekly Python Exercise A2 (functions + modules for beginners) closes today

If you are a relative beginner to Python, and want to improve your understanding of functions and modules, then there’s no better way to do so than practice.

Weekly Python Exercise provides you with that practice, with a family of six 15-week courses. In each course, you get a question on Tuesday, the answer on Monday, discussion among your cohort in our private forum, and live, monthly office hours.

And today’s the last day to sign up for the latest cohort for beginners, with an emphasis on functions and modules.

  • Do you have to check on Stack Overflow every time you write a Python function? Then this cohort of WPE is for you.
  • Do you want to have a better understanding of how scoping — local vs. global vs. builtins — works in Python? Then this cohort of WPE is for you.
  • Are you confused between *args and **kwargs, and want to know how and when to use them, without using Google? Then this cohort of WPE is for you.
  • And the Python standard library, which comes with the language — how familiar are you with the most common modules that come with the language? If you want to better understand how to use them, then this cohort of WPE is for you.
  • Finally, if you’ve wanted to write modules and use them in your code, so that you can reuse functionality across programs, then this cohort of WPE is for you.

Hundreds of previous participants in Weekly Python Exercise have improved their Python fluency, and gotten better at their jobs as a result. If you also want to improve your Python skills, then WPE is a great way to do it.

Want to learn more, or to sign up? Check out Weekly Python Exercise at https://WeeklyPythonExercise.com/ . And if you have questions? Just e-mail me, at reuven@lerner.co.il.

But don’t delay, because today (Friday) is the last day to join! I’ll be running more cohorts of WPE this year, but this particular one (A2) won’t run again until 2020.


“Python Workout” is Manning’s Deal of the Day!

If you’ve just finished a Python course or book, then you might feel a bit nervous about your Python knowledge. You might be wondering how you can become a master Python developer, solving problems without turning to Stack Overflow every few minutes.

The good news is that you can improve! But getting better at Python means practice, practice, and more practice. Just like everything else in life.

My new book, “Python Workout,” has 50 short Python challenges designed to help you become a more fluent Python developer. And today, it’s Manning’s “Deal of the day,” at 50% off its normal price!

Just go to https://www.manning.com/dotd and get 50% off “Python Workout,” as well as other Manning books.

There’s still time to join Weekly Python Exercise

Another cohort of Weekly Python Exercise starts next week! This time, it’s course A2 — for beginners, focusing on functions and modules.

Registration closes on Friday. So if you want to level up your Python skills, you should check out, and register with, Weekly Python Exercise sooner rather than later.

This cohort of WPE is for you, if:

  • You’re a bit shaky on the difference between *args and **kwargs, and when to use them
  • You don’t understand why mutable defaults are a bad things
  • You don’t know what the “global” keyword does, or why you should (or shouldn’t) use it
  • You want to create new Python modules, but aren’t sure where to start
  • You would like to be more familiar with the builtin Python standard library

The best way to learn is through a combination of practice and interactions with others — and that’s what WPE provides.

Weekly Python Exercise is the best way I know of for Python developers to improve their skills, become more fluent, and get better jobs. Join WPE, and you’ll have access not only to 15 weeks of problems and solutions, but also to a community of peers, and to monthly office hours with me.

Want to learn more? Just go to https://WeeklyPythonExercise.com/. Or if you have questions, e-mail them to me, at reuven@lerner.co.il.


Making your Python decorators even better, with functool.wraps

The good news: I gave a talk on Friday morning, at PyCon 2019, called “Practical decorators.”

The better news: It was a huge crowd, and people have responded very warmly to the talk. Thanks to everyone at PyCon who came to talk to me about it!

However: Several people, at the talk and afterwards, asked me about “functool.wraps“.

So, please think of this post as an addendum to my talk.

Let’s assume that I have the same simple decorator that I showed at the top of my talk, “mydeco”, which takes a function’s output and puts it into a string, followed by three exclamation points:

def mydeco(func):
def wrapper(*args, **kwargs):
return f'{func(args, **kwargs)}!!!'
return wrapper

Let’s now decorate two different functions with “mydeco”:

@mydeco
def add(a, b):
'''Add two objects together, the long way'''
return a + b
@mydeco
def mysum(*args):
'''Sum any numbers together, the long way'''
total = 0
for one_item in args:
total += one_item
return total

What happens when I run these functions? They do what we would expect:

>>> add(10, 20)
'30!!!'
>>> mysum(10, 20, 30, 40, 50)
'150!!!

Fantastic! We get each function’s result back, as a string, with the exclamation points. The decorator worked.

But there are a few issues with what we did. For example, what if I ask each function for its name:

>>> add.__name__
'wrapper'
>>> mysum.__name__
'wrapper'

The __name__ attribute, which gives us the name of a function when we define it, now reflects the returned internal function, “wrapper”, in our decorator. Now, this might be true, but it’s not helpful.

It gets even worse if we ask to see the docstring:

>>> help(add)
Help on function wrapper in module __main__:
wrapper(*args, **kwargs)
>>> help(mysum)
Help on function wrapper in module __main__:
wrapper(*args, **kwargs)

In other words: We are now getting the docstring and function signature of “wrapper”, the inner function. And this is a problem, because now someone cannot easily find out how our decorated function works.

We can solve this problem, at least partially, by assigning to the __name__ and __doc__ attributes in our decorator:

def mydeco(func):
def wrapper(*args, **kwargs):
return f'{func(args, **kwargs)}!!!'
wrapper.__name__ = func.__name__
wrapper.__doc__ = func.__doc__
return wrapper

If we use this version of the decorator, then each time we return “wrapper” from our decorator, then we’re doing so after first assigning the original function’s name and docstring to it. If we do this, then things will work the way we want. Mostly:

>>> help(add)
Help on function add in module __main__:

add(*args, **kwargs)
Add two objects together, the long way
>>> help(mysum)
Help on function mysum in module __main__:

mysum(*args, **kwargs)
Sum any numbers together, the long way

The good news is that we’ve now fixed the naming and the docstring problem. But the function signature is still that super-generic one, looking for both *args and **kwargs.

The solution, as people reminded me after my talk, is to use functools.wraps. It’s designed to solve precisely these problems. The irony, of course, is that it might make your head spin a bit more than decorators normally do, because functools.wraps is … a decorator, which takes an argument! Here’s how it looks:

from functools import wraps

def mydeco(func):
@wraps(func)
def wrapper(*args, *kwargs):
return f'{func(args, **kwargs)}!!!'
return wrapper

Notice what we’ve done here: We have used the “functool.wraps” decorator to decorate our inner function, “wrapper”. We’ve passed it an argument of “func”, the decorated function passed to “mydeco”. By applying this “wraps” decorator to our inner function, we copy over func’s name, docstring, and signature to our inner function, avoiding the issues that we had seen before:

>>> help(add)
Help on function add in module main:
add(a, b)
Add two objects together, the long way

>>> help(mysum)
Help on function mysum in module main:
mysum(*args)
Sum any numbers together, the long way

So, to answer the questions that I got after my talk: Yes, I would definitely recommend using functool.wraps! It costs you almost nothing (i.e., one line of code), and makes your decorated function work more normally and naturally. And I’m going to try to find a way to squeeze this recommendation into future versions of this talk, as well.

Get code + slides from my “Practical Decorators” talk from Euro Python / PyCon 2019

I presented my “Practical Decorators” talk twice this year — once at PyCon 2019 in Cleveland, and again at EuroPython 2019 in Basel. Here is the video of my presentation in Cleveland:

If you want to get the PDF of my slides, as well as the Python code that I showed, then just enter your e-mail address here. I’ll send you a link to the zipfile that you can download.

Get the bonus content: Practical Decorators — code and slides

Thanks for your interest!


>