Level 2 · Python Fluency: Idiomatic Python

Level 1 wrote programs out of the basic parts: loops, conditions, functions and collections. Those parts can express anything, but they often need more lines than the idea deserves. Python has a small set of constructs made for the patterns that come up again and again: taking part of a sequence, walking two lists side by side, building a new list from an old one, finding the best item by some measure, checking a condition for every item. Code that uses them is called idiomatic: it reads the way an experienced Python programmer would write it.

Idiomatic does not mean clever. Each construct in this lesson replaces a hand-written loop with something shorter and harder to get wrong: there is no index to start in the wrong place, no flag variable to forget to reset, no trailing comma to strip off. Many are also faster, because the loop runs inside Python's own C code instead of line by line.

You will use every one of these in the other paths. A dot product in Machine Learning is sum(a * b for a, b in zip(u, v)); a variance in Statistics is a comprehension over deviations from the mean; a train/test split is two slices; a nearest-neighbour prediction is min with a key. Each section starts from the plain version, so you can see exactly what the idiom saves.

Slicing

You have a list of labelled examples and want the first 80% for training and the rest for testing, as the Machine Learning path does. A first version copies elements one index at a time:

data = [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
cut = int(len(data) * 0.8)
train, test = [], []
for i in range(len(data)):
    if i < cut:
        train.append(data[i])
    else:
        test.append(data[i])
print(train, test)

[10, 11, 12, 13, 14, 15, 16, 17] [18, 19]

A slice data[start:stop] takes the items from start up to, but not including, stop, and either end can be left out. Because the stop is excluded, data[:cut] and data[cut:] fit together exactly, with no overlap and no gap:

data = [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
cut = int(len(data) * 0.8)
train, test = data[:cut], data[cut:]
print(train, test)
print(data[-3:])        # the last three
print(data[::2])        # every second item
print(data[::-1])       # reversed
print(data[100:])       # out of range is simply empty

[10, 11, 12, 13, 14, 15, 16, 17] [18, 19]
[17, 18, 19]
[10, 12, 14, 16, 18]
[19, 18, 17, 16, 15, 14, 13, 12, 11, 10]
[]

Slices work on strings and tuples too, and a slice always makes a new sequence, so data[:] is a quick copy. That is convenient but not free: a slice copies every element it contains. A moving average over a window of k readings (a common smoothing step in Statistics) can be written with a slice per position, which is clear and fine for small data:

readings = [3, 5, 4, 8, 6, 7]
k = 3
means = []
for i in range(len(readings) - k + 1):
    window = readings[i:i + k]
    means.append(sum(window) / k)
print(means)

[4.0, 5.666666666666667, 6.0, 7.0]

Each window is copied and summed, so the cost is n × k. The Algorithms path's sliding-window technique keeps a running sum instead (add the item entering, subtract the one leaving), which is n steps whatever k is. Reach for the slice first; switch when k is large.

Common mistakes: expecting data[2:5] to include index 5; forgetting that s[::-1] makes a full copy (fine for short strings, wasteful in a loop); slicing inside a loop over a long list without noticing the hidden copy; and assuming a slice raises an error when out of range (it quietly returns fewer items, which can hide a bug).

Practice: Train, Validation and Test, Unwinding the Scytale

Unpacking

A list of points, each an (x, y) pair, and you want the one furthest to the right. A first version reaches into each tuple by position:

points = [(2, 5), (7, 1), (4, 4)]
best = points[0]
for p in points:
    if p[0] > best[0]:
        best = p
print("x =", best[0], "y =", best[1])

x = 7 y = 1

p[0] and p[1] say nothing about what they hold. Unpacking assigns the parts of a sequence to names in one step, and it works anywhere a name is assigned, including the loop variable:

points = [(2, 5), (7, 1), (4, 4)]
for x, y in points:
    print(f"x={x} y={y}")
x, y = max(points)
print(x, y)

x=2 y=5
x=7 y=1
x=4 y=4
7 1

(max(points) works because tuples compare first by their first item; the next sections show how to choose the comparison yourself.)

The form you will use most is updating two variables at once. Python evaluates the whole right-hand side before assigning anything, so the Fibonacci step needs no temporary variable:

a, b = 0, 1
for _ in range(8):
    a, b = b, a + b
print(a)

21

A starred name collects "everything else" into a list, which is handy when a line of data has a label followed by a variable number of values. The name _ is the convention for a value you do not need:

first, *rest = [4, 8, 15, 16, 23]
print(first, rest)
*init, last = "abc"
print(init, last)
name, *scores = "ana 7 9 8".split()
print(name, scores)
_, y = (3, 9)
print(y)

4 [8, 15, 16, 23]
['a', 'b'] c
ana ['7', '9', '8']
9

Common mistakes: unpacking into the wrong number of names (x, y = (1, 2, 3) raises ValueError: too many values to unpack); writing a = b then b = a + b on separate lines for the Fibonacci step, which uses the new a; and forgetting that a starred name always gives a list, even when it collects nothing.

Practice: Packets That Do Not Add Up, Foxes and Rabbits

Comprehensions

To compute a variance, the Statistics path first finds each value's squared distance from the mean. A first version builds that list with a loop:

data = [4, 8, 6, 9, 3, 6, 7, 5]
mean = sum(data) / len(data)
sq_dev = []
for x in data:
    sq_dev.append((x - mean) ** 2)
print(sq_dev)
print(sum(sq_dev) / len(data))

[4.0, 4.0, 0.0, 9.0, 9.0, 0.0, 1.0, 1.0]
3.5

Three lines (create an empty list, loop, append) express one idea: "the squared deviation of each x". A list comprehension writes that idea as a single expression, in the same order you would say it:

data = [4, 8, 6, 9, 3, 6, 7, 5]
mean = sum(data) / len(data)
sq_dev = [(x - mean) ** 2 for x in data]
print(sq_dev)
print(sum(sq_dev) / len(data))

[4.0, 4.0, 0.0, 9.0, 9.0, 0.0, 1.0, 1.0]
3.5

The comprehension is shorter, there is no half-built list visible in the middle of the code, and it runs faster than the equivalent append loop. An if at the end keeps only some items, and the same syntax with braces builds a dictionary or a set:

v = [3.0, 4.0]
length = sum(x * x for x in v) ** 0.5
unit = [x / length for x in v]            # normalise a vector (Machine Learning)
print(unit)

temps = [21, -3, 15, -8, 30]
print([t for t in temps if t < 0])        # filter

words = ["pear", "fig", "banana"]
print({w: len(w) for w in words})         # dictionary comprehension
print(sorted({len(w) for w in words}))    # set comprehension

grid = [[r * 3 + c for c in range(3)] for r in range(2)]
print(grid)

[0.6, 0.8]
[-3, -8]
{'pear': 4, 'fig': 3, 'banana': 6}
[3, 4, 6]
[[0, 1, 2], [3, 4, 5]]

The first line of that example uses a comprehension with round brackets inside sum(...). That is a generator expression: it produces the values one at a time instead of building a list, so sum(x * x for x in v) uses almost no memory even if v has millions of items. Level 3 explains how that works.

From maths notation to comprehensions. The variance at the start of this section is \frac{1}{n}\sum_{i=1}^{n}(x_i - \bar{x})^2, where \bar{x} is the mean, and sum((x - mean) ** 2 for x in data) / len(data) is the same thing in Python. Most formulas translate this way. The index under the symbol becomes the for, what the index runs over becomes the iterable, and the expression after the symbol goes in front of the for:

import math
n = 4
print(sum(i ** 2 for i in range(1, n + 1)))     # 1 + 4 + 9 + 16
print(math.prod(range(1, n + 1)))               # 4! = 1 * 2 * 3 * 4
S = [3, -1, 4, -1, 5]
print({x for x in S if x > 0})
a = [[1, 2], [3, 4]]
print(sum(a[i][j] for i in range(2) for j in range(2)))

30
24
{3, 4, 5}
10

Two details catch people out. Maths usually counts from 1 and Python from 0, so x_1, \dots, x_n are xs[0] to xs[n - 1]; when a formula just means "over all the items", loop over the items (for x in xs) and the index question disappears. And the condition after the colon in set notation becomes the if at the end of the comprehension.

A comprehension is for building a collection. If a loop exists to print, append to some other list or change state, keep it as a loop. And if a comprehension needs two fors and two ifs, it is usually clearer as a loop with a helper function.

Common mistakes: writing [print(x) for x in items] to get a loop on one line (it builds a useless list of Nones); putting the if in the wrong place ([x if x > 0 for x in xs] is a syntax error; a filter goes at the end, while x if c else y at the front chooses a value); and nesting comprehensions so deeply that nobody can read them.

Practice: Words as Vectors, Shrinking an Image by Pooling

enumerate

A log has one message per line and you want the line numbers of the errors. A first version loops over positions so that it knows where it is:

lines = ["ok", "ERROR disk", "ok", "ok", "ERROR network"]
for i in range(len(lines)):
    if lines[i].startswith("ERROR"):
        print(i + 1, lines[i])

2 ERROR disk
5 ERROR network

The loop needs both the position and the item, so it indexes the list on every use and adds 1 by hand for human-friendly numbering. enumerate hands you both at once, and its start argument sets the first number:

lines = ["ok", "ERROR disk", "ok", "ok", "ERROR network"]
for n, line in enumerate(lines, start=1):
    if line.startswith("ERROR"):
        print(n, line)

2 ERROR disk
5 ERROR network

enumerate also makes "where is the best item?" easy. The Algorithms path often needs the index of the maximum, not just its value:

nums = [4, 9, 2, 9, 1]
best_i = 0
for i, x in enumerate(nums):
    if x > nums[best_i]:
        best_i = i
print(best_i)
print([i for i, x in enumerate(nums) if x == 9])

1
[1, 3]

Use range(len(...)) only when you truly need positions without the items, for example to compare nums[i] with nums[j] for every pair.

Common mistakes: forgetting to unpack (for pair in enumerate(xs) gives tuples like (0, 'a')); swapping the order (enumerate gives the index first); and changing the list while enumerating it.

Practice: Where the Image Is Bright, Incidents in a Log

zip

The dot product of two vectors multiplies matching entries and adds the results. It is the most used operation in the Machine Learning path. A first version walks both lists with one index:

u = [1, 2, 3]
v = [4, 5, 6]
total = 0
for i in range(len(u)):
    total += u[i] * v[i]
print(total)

32

zip pairs up items from two (or more) sequences, position by position, so the loop can name the matching entries directly. Combined with a generator expression, the dot product becomes one line that reads like its definition:

u = [1, 2, 3]
v = [4, 5, 6]
print(sum(a * b for a, b in zip(u, v)))
print(list(zip(u, v)))

manhattan = sum(abs(a - b) for a, b in zip(u, v))
print(manhattan)

32
[(1, 4), (2, 5), (3, 6)]
9

Three more patterns come up constantly. Zipping a list with itself shifted by one gives each pair of neighbours, which is how you compute the differences between consecutive readings. dict(zip(keys, values)) builds a dictionary from two lists. And zip(*rows) turns rows into columns, because the * passes each row as a separate argument:

readings = [10, 13, 12, 18]
print([b - a for a, b in zip(readings, readings[1:])])

names = ["ana", "ben"]
ages = [31, 27]
print(dict(zip(names, ages)))

rows = [(1, 2, 3), (4, 5, 6)]
print(list(zip(*rows)))

[3, -1, 6]
{'ana': 31, 'ben': 27}
[(1, 4), (2, 5), (3, 6)]

Common mistakes: zip stops at the shortest input without any warning, so a missing feature value silently drops a whole column from a dot product. When the lists must be the same length, write zip(u, v, strict=True), which raises ValueError if they are not:

try:
    print(sum(a * b for a, b in zip([1, 2, 3], [4, 5], strict=True)))
except ValueError:
    print("lengths differ")

lengths differ

Also remember that zip returns an iterator that can be used only once (Level 3 explains why); wrap it in list(...) if you need to look at it twice.

Practice: Mean of Every Column, The First Line That Changed

Sorting and choosing with key=

Sort a list of words by length. A first version builds (length, word) pairs, sorts the pairs (tuples compare by their first item) and then throws the lengths away:

words = ["banana", "fig", "cherry", "kiwi"]
pairs = [(len(w), w) for w in words]
pairs.sort()
print([w for _, w in pairs])

['fig', 'kiwi', 'banana', 'cherry']

This works, but it is three steps for one idea, and it quietly sorts equal lengths alphabetically, which you may not have wanted. The key argument does the same thing in one step: sorted calls the key function on each item and orders the items by the results, keeping ties in their original order:

words = ["banana", "fig", "cherry", "kiwi"]
print(sorted(words, key=len))
print(sorted(words, key=len, reverse=True))
print(sorted(["b", "A", "c"], key=str.lower))

['fig', 'kiwi', 'banana', 'cherry']
['banana', 'cherry', 'kiwi', 'fig']
['A', 'b', 'c']

The key can be any function of one argument, including one you write. For short keys it is common to use a lambda, a small unnamed function written in place (Level 3 looks at these properly). Returning a tuple sorts by several fields, and negating a number reverses just that field:

people = [("ana", 31), ("ben", 27), ("chen", 31)]
def by_age(person):
    return person[1]
print(sorted(people, key=by_age))
print(sorted(people, key=lambda p: (-p[1], p[0])))   # oldest first, then by name

[('ben', 27), ('ana', 31), ('chen', 31)]
[('ana', 31), ('chen', 31), ('ben', 27)]

min and max take the same key, which is the idiomatic way to find "the best item by some measure" without sorting everything. Here is the heart of the nearest-neighbour classifier from the Machine Learning path: the training example whose point is closest to the query.

import math
train = [((1, 1), "red"), ((5, 4), "blue"), ((2, 3), "red"), ((6, 6), "blue")]
query = (5, 5)
point, label = min(train, key=lambda ex: math.dist(ex[0], query))
print(point, label)
print(max(["pear", "fig", "banana"], key=len))

(5, 4) blue
banana

min with a key is a single O(n) pass; sorting to take the first item would be O(n log n) and build a whole new list.

Common mistakes: calling the key function instead of passing it (key=len() is an error; write key=len); min or max on an empty list, which raises ValueError (pass default= if the list may be empty); and trying -name to reverse a string field (strings cannot be negated; use reverse=True or sort twice, relying on stability).

Practice: Pairing Names with Scores, The Best Flight on Offer

any and all

Binary search only works on sorted data, so a careful function checks first. A first version uses a flag variable:

def is_sorted(nums):
    ok = True
    for i in range(len(nums) - 1):
        if nums[i] > nums[i + 1]:
            ok = False
    return ok

print(is_sorted([1, 3, 3, 7]), is_sorted([1, 5, 2]))

True False

The flag has to be set up before the loop and read after it, and this version keeps checking even after it has found a problem. all says "true for every item" directly, and any says "true for at least one". Both stop at the first item that settles the answer, so they are as fast as a loop with break:

def is_sorted(nums):
    return all(a <= b for a, b in zip(nums, nums[1:]))

print(is_sorted([1, 3, 3, 7]), is_sorted([1, 5, 2]))

temps = [21, 19, -2, 25]
print(any(t < 0 for t in temps))
print(all(isinstance(t, int) for t in temps))
print(any([]), all([]))

True False
True
True
False True

The last line is worth remembering: any of nothing is False (no item was true) and all of nothing is True (no item was false). This matches the logic of "every" and "some" in mathematics, and it is usually what you want: an empty list is sorted.

any and all are at their best with a generator expression inside, as above, because the generator produces values only as they are needed. With a list comprehension inside (any([t < 0 for t in temps])) every comparison is made first, and the early exit is lost.

Common mistakes: forgetting that all([]) is True, so an empty dataset passes every validation check (test for emptiness separately if that matters); writing any(temps < 0), which compares a list with a number; and relying on truthiness when zero is a valid value (all(scores) is false if any score is 0).

Practice: From Survey Records to X and y, Is It a Valid Transition Table?, Models Worth Keeping

Built-in functions

A line of text holds lap times in seconds, and you want the total time as minutes and seconds. A first version does every step by hand:

line = "74 81 69 90"
times = []
for part in line.split():
    times.append(int(part))
total = 0
for t in times:
    total += t
minutes = 0
seconds = total
while seconds >= 60:
    seconds -= 60
    minutes += 1
print(f"{minutes}:{seconds:02d}")

5:14

Every one of those loops is a built-in function. map(int, ...) applies int to each piece, sum adds them up and divmod returns the quotient and remainder together:

line = "74 81 69 90"
times = list(map(int, line.split()))
minutes, seconds = divmod(sum(times), 60)
print(f"{minutes}:{seconds:02d}")
print(min(times), max(times), len(times))

5:14
69 90 4

The built-ins are written in C, tested by millions of programs and named after what they do, so they are both faster and easier to read than the loops they replace. The ones you will use most, beyond len, min, max and sum:

print(abs(-3.5), round(2.675, 2), round(7.5), round(8.5))
print(list(reversed([1, 2, 3])))
print(list(filter(None, [0, 3, "", "a", None])))   # keep the truthy items
print(divmod(-7, 2))

3.5 2.67 8 8
[3, 2, 1]
[3, 'a']
(-4, 1)

Two of those results surprise people. round(2.675, 2) gives 2.67 because 2.675 cannot be stored exactly and is really a hair below it. And round sends exact halves to the nearest even number (round(7.5) and round(8.5) are both 8), which avoids a bias upwards when many values are rounded. That is the kind of detail the Statistics path cares about. divmod(-7, 2) follows floor division, so the quotient rounds down to -4 and the remainder is positive.

map and filter are handy with a function that already exists (map(int, parts), map(str.strip, lines)). When you would need a lambda, a comprehension is usually clearer: [x * 2 for x in xs] beats list(map(lambda x: x * 2, xs)).

Common mistakes: naming a variable sum, max, list or min, which hides the built-in for the rest of the program (sum = 0 and then sum(xs) fails with TypeError: 'int' object is not callable); forgetting that map, filter and reversed return one-use iterators rather than lists; and using sum to join strings (use str.join, next section).

Practice: Ticket Numbers to Seats, A Column with Gaps

Joining strings

Turn a row of values into one line of a comma-separated file. A first version adds each value and a comma to a growing string, then has to remove the extra comma at the end:

row = ["ana", 31, 7.5]
line = ""
for value in row:
    line += str(value) + ","
line = line[:-1]
print(line)

ana,31,7.5

The trailing-separator fix is easy to forget, and the str(...) calls clutter the loop. There is also a cost: strings cannot be changed, so each += may build a new string and copy everything so far, which makes the loop O(n²) for long outputs. str.join is called on the separator, takes the pieces and puts the separator only between them, in one pass:

row = ["ana", 31, 7.5]
print(",".join(str(v) for v in row))
print(" | ".join(["a", "b", "c"]))
print("".join(reversed("stressed")))
print("-".join([]))            # nothing to join: the empty string

ana,31,7.5
a | b | c
desserts

For text made of fixed parts with a few values in them, an f-string is the natural tool. Together they build a neat table without any +:

rows = [("ana", 7.5), ("ben", 9.25), ("chen", 8.0)]
lines = [f"{name:<6}{score:>6.2f}" for name, score in rows]
print("\n".join(lines))
print(f"{len(rows)} students, mean {sum(s for _, s in rows) / len(rows):.2f}")

ana     7.50
ben     9.25
chen    8.00
3 students, mean 8.25

:<6 pads on the right to 6 characters and :>6.2f right-aligns a two-decimal number, so the columns line up. The pattern "build a list of lines, then join them with "\n" once" is the idiomatic way to produce any multi-line output.

Common mistakes: passing numbers to join (it only accepts strings, so convert with str(v) first); calling it the wrong way round (words.join(" ") fails because join belongs to the separator); and building long text with += in a loop.

Practice: Writing a Spreadsheet Export

Practice

These problems are required on this path at this level; the path adds one problem from each of the level's topics, and one for each construct the others do not use, picked for you and kept.

• Pairing Names with Scores
• The Best Flight on Offer
• Is It a Valid Transition Table?
• Train, Validation and Test
• Packets That Do Not Add Up
• Unwinding the Scytale

Checklist

• Split a list into parts with slices, reverse or step through it, and know that each slice is a copy.
• Unpack tuples in assignments and loops, swap with a, b = b, a, and collect the rest with *rest.
• Replace a build-a-list loop with a list, dictionary or set comprehension, and use a generator expression inside sum, any and all.
• Loop with enumerate when you need positions and items, and with zip when walking two sequences together (with strict=True when lengths must match).
• Sort, and pick the best item, with key=, including tuple keys for several fields.
• Test "every" and "some" with all and any, and explain why all([]) is True.
• Use sum, divmod, abs, round, map, and reversed instead of hand-written loops, and never shadow their names.
• Build output with str.join and f-strings, aligned with format specifications, instead of += in a loop.