It’s my annual birthday sale!

Today (Sunday, July 14th) is my birthday. And as happens every year, I’m celebrating with huge discounts on all of my online courses and books. I’m now 49, so you can get 49% off of anything in my online store. But as with birthdays, this massive sale won’t last long — it ends Monday evening.

Just use the coupon code BD2019 at my online store (https://store.lerner.co.il/), for 49% off of the following:

This is the biggest discount I’ve ever given on these courses… so enjoy the discount, celebrate my birthday with me, and improve your understanding of Python, data science, and Git while you can.

Questions? Just e-mail me at reuven@lerner.co.il.

Registration for Weekly Python Exercise closes today!

If you’ve been using Python for at least six months, but still find yourself searching on Stack Overflow and Google for answers to your problems — you’re not alone.

The good news is that the solution to your problem, to greater Python fluency, is easy: Practice, practice, and more practice.

My new advanced-level cohort of Weekly Python Exercise starts on Tuesday (July 2nd). Today is the deadline to sign up! Just go to https://WeeklyPythonExercise.com/ to learn more and start improving your Python skills.

2

Announcing: Python standard library, video explainer

A month or two ago, I saw an online quiz that caught my eye: How much of the Python standard library do you know?

Now, the “standard library” is the collection of modules and packages that come with Python. It constitutes the “batteries” that “batteries included” refers to in the Python world. And the standard library is big, with about 300 modules, each of which contains functions, classes, and values.  Knowing the standard library, and how to use it, is essential to productive use of Python.

And yet, a large number of the people responding indicated that they knew very little of the standard library.  Which makes sense, given that each of us tends to focus on what’s important to our jobs.

Indeed, I’ve been using Python for a long time.  And I had to admit that there are large parts of the standard library with which I’m totally unfamiliar.  I’m sure that there are gems (no Ruby pun intended) in there that I could make use of in my work, if I only knew about them.

Image

I’ve thus decided to try an experiment, namely to walk through the entire standard library (or as much as I can physically, humanly do) in an open-ended YouTube video series.  I’ve already uploaded a number of the videos, and I will be uploading a few new ones every week.  I’m starting with the builtin types, walking through each of their methods — but I’ll then proceed to the other and lesser-known modules.

This is all new content, certainly overlapping my previous writing and courses to some degree, but made new for this series.  It’s less formal than my courses, without any exercises or background theory connecting it all together.  And yet, I’ve already learned about a number of methods and techniques — and I assume that the same will be true for you.

This video explainer is on YouTube, so you can watch it easily by subscribing to my channel there, at https://www.youtube.com/reuvenlerner.  This is a long-term project, but one that I hope will be helpful and of use to you and to the entire Python community!  Subscribe to my channel, and you’ll get updated whenever I add new videos.

2

Want to level up your Python? Join Weekly Python Exercise, starting July 2nd

Let’s face it: Stack Overflow has made developers’ lives easier. Almost every time I have a question, I find that someone on Stack Overflow has asked it, and that people have answered it, often in great detail.

I’m thus not against Stack Overflow, not by a long shot. But I have found that many Python developers visit there 10 or even 20 times a day, to find answers (and even code) that they can use to solve their problems.

Can you work this way? Yes, and many people do — but it’s not the ideal, which would be to have greater Python fluency. If you could know how to solve the problem without looking it up, you would be able to accomplish more in less time.

Moreover, achieving Python fluency means that when you do need to search, you’ll do it better, more quickly, and more accurately than before. You’ll be able to design larger and more complex systems, using your greater understanding of Python’s functionality and data structures to construct more sophisticated systems.

How can you achieve such fluency? Practice, practice, and more practice.

Weekly Python Exercise is a family of 15-week courses, each of which gives you that practice, along with community discussion and live office hours. As you work your way through WPE’s exercises, you’ll get a better and deeper understanding of how to write Pythonic code, how to use the standard library, and what packages on PyPI are worthwhile.

Here’s what one student has to say:

From my perspective, WPE has given me a broad sense of what the Python language is all about. WPE has not just taught me syntax but has given context with respect to what types of tasks the language is meant to solve… Reuven’s explanations per task really do bring each task full circle for understanding and approach.

A new advanced-level cohort, aimed at people with at least six months of Python experience,starts on July 2nd. You can sign up at any time until then — but avoid the last-minute penalty by joining before Friday, June 28th. During this cohort, we’ll explore such topics as iterators, generators, decorators, functional programming, and threads. Every question comes with a test written in “pytest”. And you’ll be able to exchange code, ideas, and suggestions with others in your cohort via our private, exclusive forum.

If you want to solve bigger and better problems with Python, if you want to take your Python to the next level, and if you want to push yourself to learn new topics, then you should join WPE.

Learn more about Weekly Python Exercise, and how it’ll help you to become a better Python programmer — doing more in less time, and doing it better. You can learn more at https://WeeklyPythonExercise.com.

Early-bird pricing for Weekly Python Exercise ends today!

Just a reminder: Registration for the advanced (B2) cohort of Weekly Python Exercise, which will begin on July 2nd, will remain open for the next two weeks. BUT early-bird pricing ($80 for 15 weeks of Python exercises, solutions, and community) ends today — Tuesday, June 18th.

If you want to sharpen your Python skills, then there’s no better way to do that than Weekly Python Exercise.

And hey, if you’re going to sharpen your skills, why not do it at a discount? As of tomorrow, you’ll have to pay more for the same course.

Learn more at https://WeeklyPythonExercise.com/ . 100% money-back guarantee if you aren’t satisfied — but I’m sure you’ll learn so much, and be able to solve so many new problems, that you won’t want to do that.

6

Understanding Python assignment

Here’s a quick question I often ask students in my Python classes:

>>> x = 100
>>> y = x
>>> x = 200

After executing the above code, what is the value of y?

The answer:

>>> print(y)
100

Many of my students, especially those with a background in C, are surprised. Didn’t we say that “y = x”? Thus, shouldn’t a change in x be reflected by a similar change in y?

Obviously not. But why is this the case?

Assignment in Python means one thing, and one thing only: The variable named on the left should now refer to the value on the right.

In other words, when I said:

y = x

Python doesn’t read this as, “y should now refer to the variable x.” Rather, it read it as, “y should now refer to whatever value x refers to.”

Because x refers to the integer 100, y now refers to the integer 100. After these two assignments (“x = 100” and “y = x”), there are now two references to the integer 100 that didn’t previously exist.

When we say that “x = 200”, we’re removing one of those references, such that x no longer refers to 100. Instead, x will now refer to the integer 200.

But y’s reference remains in place to where it was originally pointing, to 100. And indeed, the only way to change what y is referring to is via … assignment.

Think of this as assignment inertia: Without a new and explicit assignment, a variable will continue to refer to whatever it referred to previously.

Thus, while Python does have references (i.e., variables pointing to objects), it doesn’t have pointers (i.e., variables pointing to other variables). That’s a big difference, and one that makes the language easier to understand. But references can still be a bit tricky and confusing, especially for newcomers to the language.

Remember also that in an assignment, the right side is evaluated before the left side. By the time the left-hand-side is being assigned, any variables on the right-hand-side are long gone, replaced by the final value of the expression. For example:

>>> a = 10
>>> b = 20
>>> c = 30
>>> d = a + b * c

When we assign a value to the variable “d” above, it’s only after Python has evaluated “a + b * c”. The variables are replaced by the values to which they refer, the operations are evaluated, and the final result (610) is then assigned to “d”. “d” has no idea that it was ever getting a value from “a”, “b”, or “c”.

Reminder: Early-bird pricing for Weekly Python Exercise ends tomorrow

This is just a quick reminder that if you want to join the advanced cohort of Weekly Python Exercise starting July 2nd, you should do it by tomorrow (Tuesday, June 18th).

Don’t miss this opportunity to improve your Python coding skills! We’ll be talking about iterators, generators, decorators, threads, and functional programming, and helping you to improve your skills.

Questions? Just e-mail me at reuven@lerner.co.il. But hurry, before the price goes up!

1

Playing with Python strings, lists, and variable names — or, a complex answer to a simple question

I recently received a question from a reader of my “Better developers” list. He asks:

Is there any way to turn a str type into a list type? For example, I have a list of elements, and want to turn that element into a separate list. For example, if I have

test = ['a', 'b', 'c']

I want the output to be

a=[], b=[], c=[]

One of the mantras of Python is that there should be one, and only one, way to do something. Reality has a way of being more complex than that, though, and in this particular case, the problem that my reader described in words and what he put in code weren’t exactly the same thing. (Which is a common problem in the professional software world — the specifications say one thing, but the client’s intentions say another.)

Let’s start with what my reader says he wants to do, and then get to what he actually seems to want:

He says that he wants to turn a string into a list. Well, there are a few ways to do that. The easiest is to use the “list” class, and apply it to a function:

He says that he wants to turn a string into a list. Well, there are a few ways to do that. The easiest is to use the “list” class, and apply it to a function:

>>> s = 'abc'
>>> mylist = list(s)
>>> mylist
['a', 'b', 'c']         

In such a case, the “list” class (which can be called, like a function, and is thus known as a “callable” in the Python world) iterates over the elements of our string. Each element is turned into a separate element in a new list that it returns.

This is fine if you want to create a new list with the same number of elements as there are characters in the string. After all, both strings and lists are Python sequences; when you create a list in this way, based on a string, you’ll find that the new list’s length and elements are identical. So s[0] and mylist[0] will return the same result, as will “len(s)” and “len(mylist)” even though “s” and “mylist” are different types.

Another way to create a list from a string is via the “str.split” method. I use this method all the time, especially when taking input from a user and iterating over the words, or fields, that the user provides. For example:

>>> words = 'here are some words'
>>> words.split(' ')
['here', 'are', 'some', 'words']

The result of “str.split” is always a list of strings. And as you can see in the above example, we can tell “str.split” what string should be used as a field delimiter; “str.split” removes all occurrences of that string, returning a list of strings.

What happens if our string is a bit weird, though, such as:

>>> words = 'here    are some     words'

Now we’re going to get an equally weird result:

>>> words.split(' ')
['here', '', '', '', 'are', 'some', '', '', '', '', 'words']

This happens because “str.split” has taken our instructions very literally, as computers do: Whenever you encounter a space character, create a new element in the output list. However, this is rarely the solution that you want, and thus “str.split” has a great default: If you don’t pass anything (or pass “None” explicitly), then any length of whitespace characters will be treated as a single delimiter. Which means that we can say:

>>> words = 'here    are some     words'
>>> words.split()
['here', 'are', 'some', 'words']

This is quite useful… and yet, while this is how I interpreted the question I got, it’s not what the user wants.

Rather, what he seems to want is to create new variables based on the elements of the string. So if the string is “abc”, then we want to create new variables “a”, “b”, and “c”, each of which references an empty list.

This is certainly possible, but I’ll admit it’s a bit odd. However, it gives us a chance to delve into some of Python’s more rarely used capabilities. (At least, I almost never use them — maybe other people are different!)

My first reaction to creating variables dynamically is to say, “No, you don’t really want to do that,” and to suggest that we create a dictionary, instead. You can think of a dict as your own private namespace, one which can’t and won’t interfere with the variables created elsewhere.

We could create an empty dictionary, and then iterate over the string, adding new key-value pairs to it, with each value being an empty list:

>>> for one_letter in 'abc':
        d[one_letter] = []

>>> d
{'a': [], 'b': [], 'c': []}

There is, however, a better way to do what we did here, and that is by using the “dict.fromkeys” class method. This is a great shortcut to creating a dictionary whose keys are known but whose values aren’t, at least not at the start. So we can say:

>>> dict.fromkeys('abc')
{'a': None, 'b': None, 'c': None}

As you can see, the value associated with each key here is “None”. We don’t want that; instead, we want to have an empty list. So we can pass an empty list as a second, optional argument to “dict.fromkeys”:

>>> dict.fromkeys('abc', [])
{'a': [], 'b': [], 'c': []}

However, you should be a bit nervous before working with the dictionary I’ve created here, because every single one of the values now refers to the same list! For example:

>>> d = dict.fromkeys('abc', [])
>>> d
{'a': [], 'b': [], 'c': []}
>>> d['a'].append(1)
>>> d['b'].append(2)
>>> d['c'].append(3)
>>> d
{'a': [1, 2, 3], 'b': [1, 2, 3], 'c': [1, 2, 3]}                

In many ways, this is similar to the problem of mutable defaults, in that we have a single value referenced in multiple places. It’s pretty obvious to experienced Python developers that this will happen, but it’s far from obvious to newcomers.

Another way to do this would be to use a dict comprehension:

>>> {one_letter : []
     for one_letter in 'abc'}
{'a': [], 'b': [], 'c': []}

“Wait,” you might be saying, “Maybe we have to worry about these lists also all referring to the same thing?”

Nope:

>>> d = {one_letter : []
         for one_letter in 'abc'}
>>> d['a'].append(1)
>>> d['b'].append(2)
>>> d['c'].append(3)
>>> d
{'a': [1], 'b': [2], 'c': [3]}         

What’s the difference between this, and our previous use of “dict.fromkeys”? The difference is that here, the “[]” empty list is evaluated anew with each iteration over the string. Thus, we get a new empty list each time. By contrast, passing the same empty list as a second argument to “dict.fromkeys” gave us the same list each time.

So if you want to use a dict — and that’s my recommendation — then you are good to go! But if you really and truly want to create variables based on the values in the string, then we’ll have to use a few more tricks.

One is to take advantage of the fact that global variables are actually stored in a dictionary. Yes, that’s right — you might think that when you write “x=100” that you’re storing things in some magical location. But actually, Python turns your variable name into a string, and uses that string as a key into a dictionary.

We don’t have direct access to this dictionary, but we can retrieve it using the “globals” builtin function. Here’s what happens when I invoke “globals” in a brand-new Python 3 interactive shell:

>>> globals()
{'__name__': 'main', '__doc__': None, '__package__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>}         

See what happens now, after I assign some variables:

>>> x = 100
>>> y = [10, 20, 30]
>>> z = {'a':1, 'b':2}
>>> globals()         
{'__name__': 'main', '__doc__': None, '__package__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, 'x': 100, 'y': [10, 20, 30], 'z': {'a': 1, 'b': 2}}         

Take a look at the end, and you’ll see our three newly assigned variables.

It turns out that we can also define (or update the values of) global variables in this way, too:

>>> globals()['x'] = 234
>>> globals()['y'] = [9,8,7,6]
>>> globals()['z'] = 'hello out there'         
>>> globals()
{'__name__': 'main', '__doc__': None, '__package__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, 'x': 234, 'y': [9, 8, 7, 6], 'z': 'hello out there'}

I don’t really recommend this in actual code, but if you’re absolutely, positively sure that you want to do this, then you can accomplish this task in the following way:

>>> for one_letter in 'abc':
    globals()[one_letter] = []         

Sure enough:

>>> x
[]
>>> y
[]
>>> z
[]

Again, you almost certainly don’t want to have this sort of code in production. But it does work, as we see here.

Something else we could do is use the “exec” function, which lets us run any string as a tiny Python program. We could thus say:

>> for one_letter in 'abc':
        exec(f'{one_letter} = []')

>>> a
[]
>>> b
[]
>>> c
[]         

As you can see, it worked: We used an f-string to create a tiny (one-statement) Python program, and then used “exec” to run it. Note that we wouldn’t be able to use the related “eval” function here, because “eval” expects to have an expression, and assignment in Python isn’t an expression.

Finally, I’d generally argue that it’s a good idea not to create or manipulate global variables whose names are created dynamically from the user’s input. It’s probably best (as I wrote above) to use a dictionary. However, if you really insist on doing this, then you should probably do it in a module.

But wait — aren’t modules normally defined in files? Yes, but you can create a module on the fly by running the “module” class, just as we did above with the “list” class. There’s just one hitch, namely that the “module” class isn’t available to us in any of the Python namespaces.

That’s OK: We can grab the class via another module (e.g., __builtins__), and then invoke it, passing it the name of the module we want to create. Then we can use the builtin “setattr” function to assign a new attribute to the module. Here’s how that would look:

>>> mymod = type(__builtins__)('mymod')
>>> for one_letter in 'abc':
setattr(mymod, one_letter, [])
>>> vars(mymod)
{'__name__': 'mymod', '__doc__': None, '__package__': None, '__loader__': None, '__spec__': None, 'a': [], 'b': [], 'c': []}         

Sure enough, we’ve managed to do it!

By the way, remember how I mentioned, all the way back, that it would probably be best to use a dictionary, rather than create actual variables? Well, as you can see here, a module is actually just a fancy wrapper around… a dictionary.

This seemingly simple question raised all sorts of interesting Python functionality, none of which (I’m guessing) was ever intended by the person who asked the question. But I hope that this has given you a glimpse into the ways in which Python has implemented, and how a dynamic language allows us to play with our environment in ways that not only stretch our minds, but sometimes even the boundaries of good taste.

1

“Python Workout” is Manning’s Deal of the Day!

I’m a firm believer in improving your Python fluency via practice, practice, and more practice. “Python Workout” is a collection of my 50 favorite exercises from my 20 years of on-site Python training at some of the world’s largest and best-known companies.

I’m delighted to announce that “Python Workout,” my book with 50 exercises to improve your Python mastery and fluency, is Manning’s “Deal of the Day.”

That means that if you buy it today (i.e., June 13th), you’ll get 50% off!

Python Workout is currently available as an online MEAP (Manning Early-Access Program) book. Chapters 1-3 are already available, and chapters 4-6 will be up within another week or two. (And I’m working hard to finish editing the remaining chapters ASAP.)

Don’t miss this chance to get lots of extra Python practice for a low price. Get the book at https://www.manning.com/dotd , but only today (Thursday, June 13th)!

1

Variables are pronouns: A simple metaphor for Python newbies

I teach about 10 different courses to companies around the world, but my favorite remains “Python for non-programmers.” Participants in this course are typically network and system administrators, support engineers, and managers who want to learn some programming skills, but don’t see themselves as programmers. Moreover, many of them took a programming course back when they were university students, and were so horrified, overwhelmed, and frustrated that they gave up. Perhaps they’re still working for a high-tech company, but they have tried to avoid programming.

But jobs increasingly require some knowledge of programming, and Python is a perfect language with which to start: The syntax is consistent, and the number of things you need to learn is relatively small in order to get up and running.

But that doesn’t mean that there’s nothing to learn. And one of the hardest ideas for people to learn is that of variables. Sure, people know about variables from when they learned algebra — but variables in programming languages aren’t exactly the same thing, even if there are similarities.

For years, I struggled to explain variables: I used the mailbox metaphor (which I learned from the wonderful “Computer Science, Logo Style“). But the mailbox model doesn’t really fit Python, so I’d end up saying, “Actually, I lied to you yesterday. Variables are actually references.” Which didn’t do much to clear things up. And when we started to talk about lists of lists, it wasn’t clear how much my explanations really helped.

Finally, I hit upon a metaphor that seems to resonate with people: Variables are pronouns. This has several advantages:

  • Everyone knows what pronouns are. So it’s easy to understand how we might use them, and how they save us time. After all, you it’s far easier to say “he” or “him,” rather than “Rufus Xavier Sarsaparilla.”
  • The notion of references emerges naturally from this description. No longer do I introduce the mailbox model, and then point out how it doesn’t really work, given how assignment in Python works.
  • It becomes obvious that a variable refers to the object to which it was most recently assigned, much as “she” refers to the most recent female to whom we referred. Just as “he” and “she” can only refer to a single object at a time, so too can variables only refer to a single object at a time.
  • It also follows that while a variable (pronoun) can only refer to a single object, an object might be referred to by several pronouns.

No metaphor is perfect, and it’s still tough for many people to wrap their heads around the idea of variables when they’re programming for the first time. But this model seems to have had the greatest success so far. If you teach Python programming, then give it a whirl, and let me know if it seems to help!

>