12

You can, but should you? Combining some of Python’s more esoteric features

Note: As usual, readers of my blog (and my students) found a simpler and more elegant solution than I did… I knew that the boolean “or” operator returned the first “True” value it encountered, but I completely forgot that if it only encounters “False” values, it returns the final one. So there was a far more elegant solution than I suggested, namely “[LIST COMPREHENSION] or None”. I’ll leave this blog post up, since it was still fun to explore these ideas… but as usual, when you get too complex in Python, you’re probably overlooking a simpler and more straightforward solution, as I did here.

A few weeks ago, I held my monthly “office hours” session for subscribers to Weekly Python Exercise. WPE students are always invited not only to ask questions about what we’re learning in the course, but also any other Python-related issue that they have encountered.

Well, someone asked quite a doozy this month: He said that he wants to use a list comprehension to create a list for a project at work. Except that if the list comprehension is empty, then he wants to get a “None” value.

In other words: He wants a list comprehension, or at the very least an expression containing one, which will either return a list (if non-empty) or “None” (if the list is empty).

In answering this question, I managed to pull together what might be the greatest collection of unreadable Python constructs in a single expression. I’m not recommending that you write this sort of code — but it does demonstrate that Python’s syntax does lend itself to all sorts of creative solutions and possibilities, if you know how to combine things.

Let’s start by pointing out that a list comprehension always returns a list. Regardless of how many elements it might contain, the result of a list comprehension is always going to be a list. For example:

>>> [x*x for x in range(5)]
[0, 1, 4, 9, 16]

>>> [x*x for x in range(0)]
[]

In both of the above cases, a list value was returned; there’s no such thing as a list comprehension that returns a non-list value. Even an empty list is a list, after all.

My student would thus need to accept that while a list comprehension could be part of the solution, it couldn’t be the entire solution. We would need something like an if-else statement. For example:

mylist = [x*x for x in range(5)]

if mylist:
    output = mylist
else:
    output = None

The above code will certainly work, and would be my preferred way to solve such a problem. But for whatever reason, my student said that we needed to use a single expression; an if-else statement wouldn’t suffice.

Fortunately, Python does offer an inline, expression version of if-else. I personally find it hard to read and understand, but it’s designed for situations like this one, in which we need a conditional expression. It looks like this:

TRUE_OUTPUT if CONDITION else FALSE_OUTPUT

In other words, it’s a one-line “if-else” expression, returning one value if the condition is met, and another value if it is not. For example:

>>> 'Yes' if True else 'No'
'Yes'
>>> 'Yes' if False else 'No'
'No'

Of course, we can have any expression that we might like. So we could say:

>>> mylist = [x*x for x in range(5)]
>>> mylist if len(mylist) > 0 else None
[0, 1, 4, 9, 16]

In other words: If “mylist” is non-empty, then we’ll get “mylist” back. Otherwise, we’ll get “None” back. And it works!

However, it’s considered un-Pythonic to check for an empty list (or any other empty data structure) by checking its length. Rather, we can check to see if it’s empty simply by putting “mylist” in an “if” statement. In a boolean context, all lists (as well as strings, tuples, and dicts) return “True” so long as they contain any values, but “False” if they’re empty. We can thus rewrite the above code as:

>>> mylist = [x*x for x in range(5)]
>>> mylist if mylist else None
[0, 1, 4, 9, 16]

This is fine, but it’s not a single expression, which was a requirement. Fortunately, we can just squish everything into a single line, replacing any reference to “mylist” with the list comprehension itself:

>>> [x*x for x in range(5)] if [x*x for x in range(5)] else None
[0, 1, 4, 9, 16]

Now, this is getting pretty ugly. Among other things, we have repeated our list comprehension twice in the same line. After all, our one-line “if-else” expression is just that, an expression, with no assignment allowed. So if we want to keep things on a single line, only using expressions, there’s no way for us to store the output from our list comprehension for later, is there?

There wasn’t. But then came Python 3.8 with the “assignment expression” operator, aka “the walrus,” which changed everything. The walrus is designed to be used in just this kind of situation. OK, maybe not quite this ugly of an expression, but it can help us to get out of such pickles.

What I can do is use the walrus operator to capture the list created by the list comprehension. We can then use the variable to which we’ve assigned our list, thus saving us from having to use the list comprehension a second time.

Note that the one-line “if-else” is confusing on several fronts, but nowhere more so than the fact that the condition (in the middle of the expression) executes first, before either of the output expressions is evaluated. This makes sense, when you think about it. but it can still be confusing to put an assignment in the middle of a line, so that it can be used at the start of the line.

So, let’s try it:

output if output := [x*x for x in range(5)] else None

This doesn’t work, and that’s because we need to put the central expression inside of parentheses to ensure that Python’s parser knows what is going on:

>>> output if (output := [x*x for x in range(5)]) else None
[0, 1, 4, 9, 16]

It worked! Thanks to a combination of the condition expression, the walrus operator, list comprehensions, and the fact that empty lists are “False” in a boolean context, we managed to get a single expression that returns the result of a list comprehension when it contains values, and “None” otherwise.

And while I would question the wisdom of having such code in an actual production system, I freely admit that there are times when such hacks are necessary. And despite the fact that Python has a more rigid syntax than many other languages, its functional parts made it possible for us to achieve our goal with only a minimum of code.

  • And it works!

    However, it’s considered un-Pythonic to check for an empty list (or any other empty data structure) by checking its length.

  • Quantum Mechanic says:

    It’s taken Python a very long time to get to the Walrus operator.

    In Perl, I’m often doing something like this. Say I’m pulling a few things out of a log file, ad hoc like, and trying to match a few different lines for different things. One might look like this:

    if (my($x) = $line =~ m/some_regex_capture_here/) {
    do_something_with($x);
    }

    Otherwise, we end up not being DRY, and get something like this:

    my $x;
    $x = $line =~ m/some_regex_capture_here/;
    if (defined($x)) {
    do_something_with($x);
    }

    So instead of mentioning $x 4 times in the latter, it’s only mentioned twice in the former. Just fewer places to mess up, and maintenance is easier.

    Of course, if we don’t need to keep the value at all, we could do this instead:

    do_something_with($line =~ m/some_regex_capture_here/);

    …and let the function deal with it (no variable names).

  • Anon says:

    I found this ambiguous:
    So there was a far more elegant solution than I suggested, namely “[LIST COMPREHENSION] or None”

    Does the part in quotes “…” refer to the far more elegant solution, or to what you suggested? It could be either.

    • reuven says:

      Ah, sorry for the ambiguity. I basically meant that if the question is “How can I either return the value of a list comprehension or None,” then you can go through all of the twists and turns that I describe in the blog post, in which the list comprehension is assigned, and we use the expression version of if-else.

      Or you can just say [LIST COMPREHENSION] or None; if the list comprehension returns an empty list, then the None is returned as the expression’s value.

  • Anon says:

    `[x for x in xs] or None` is the simplest, most readable one-liner for doing this. If the comprehension isn’t too complex, this seems fine to me.

  • Ross Vandegrift says:

    Python’s boolean operators return the last value they looked at. This provides another solution:
    “`
    output = [x*x for x in range(…)] or None
    “`

  • Tryph says:

    `[x*x for x in range(0)] or None`
    would be a more concise and readable option IMHO, and it is should be valid even in Python 2 😉

  • Mark Tolonen says:

    [1,2,3] or None => [1,2,3] vs. [] or None => None

  • Nathan says:

    One thing I’ve seen in the past that I think makes this a little more readable is using ‘or’ in the assignment:

    >>> [x*x for x in range(5)] or None

    Of course you have to understand what the or does in this case! And you also need the empty list to be falsey. So maybe it is more confusing. But it seems to be a little easier to parse, at least to me

  • Evan says:

    How about `output = [x*x for x in range(how_many)] or None`? Evaluates to the list if how_many is greater than 0, None otherwise.

  • Adam says:

    You want the ‘or’ operator:

    output = ([x*x for x in range(5)] or None) will be the list if it is not empty (True), and None if the list is empty (False).

  • Michael Barber says:

    Are you overlooking something, or am I?

    >>> [x*x for x in range(5)] or None
    [0, 1, 4, 9, 16]
    >>> [x*x for x in range(0)] or None
    >>>

  • >