Level 5 · Python Fluency: Python Engineering
By now you can write Python that is clear and idiomatic. This level is about writing Python that stays correct as it grows and as other people use it. Engineering, in this sense, is everything around the core calculation: adding behaviour such as caching or timing without touching the function itself, making sure resources and settings are always restored, saying what types a function expects, proving with tests that it works, finding out quickly why it does not, knowing which standard-library tool already solves your problem, and designing functions that are hard to call wrongly.
Two language features lead the way. Decorators wrap a function in extra behaviour with a single line, and context managers guarantee that clean-up happens with a with block. Both look like syntax at first, and both turn out to be ordinary functions and objects, built from the pieces of Levels 3 and 4 (closures, functions as values and protocols).
The rest of the level is habits. Every other path benefits from them: a memoised recursion in Algorithms is one decorator, a Statistics simulation that restores its random seed is one context manager, and a Machine Learning pipeline with type hints and a handful of tests is far easier to trust than one checked by eye.
Decorators
How many calls does the plain recursive Fibonacci function make? A first version counts them with a global variable written into the function body:
calls = 0
def fib(n):
global calls
calls += 1
return n if n < 2 else fib(n - 1) + fib(n - 2)
print(fib(20), calls)
6765 21891
The counting is now tangled up with the mathematics: to count calls of another function you must edit that one too, and to stop counting you must edit both again. A decorator is a function that takes a function and returns a new function wrapping it, so the extra behaviour lives in one place and can be added to anything. The line @count_calls above a definition is shorthand for fib = count_calls(fib):
import functools
def count_calls(func):
@functools.wraps(func) # keep the original name and docstring
def wrapper(*args, **kwargs):
wrapper.calls += 1
return func(*args, **kwargs)
wrapper.calls = 0
return wrapper
@count_calls
def fib(n):
"""The n-th Fibonacci number."""
return n if n < 2 else fib(n - 1) + fib(n - 2)
print(fib(20), fib.calls)
print(fib.__name__, fib.__doc__)
6765 21891
fib The n-th Fibonacci number.
The wrapper is a closure: it remembers func and calls it, doing its own work before or after. *args, **kwargs collect any positional and keyword arguments, so the same wrapper fits any function. The recursive calls inside fib look up the name fib, which now refers to the wrapper, so every call is counted. Without functools.wraps, the decorated function would report its name as wrapper and lose its docstring, which makes error messages and help text confusing.
21,891 calls to compute one number is the problem the Algorithms path solves with memoisation: remember each result so it is computed only once. That is exactly the kind of behaviour a decorator can add, and the standard library already has it as functools.cache. Stacking decorators applies them from the bottom up, so here the counter sees only the calls that miss the cache:
import functools
def count_calls(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
wrapper.calls += 1
return func(*args, **kwargs)
wrapper.calls = 0
return wrapper
@functools.cache
@count_calls
def fib(n):
return n if n < 2 else fib(n - 1) + fib(n - 2)
print(fib(20), fib.__wrapped__.calls)
print(fib(90))
6765 21
2880067194370816120
Twenty-one calls instead of 21,891, and fib(90), which the plain version would take thousands of years to finish, is instant. (__wrapped__ is the attribute functools.wraps adds to reach the function underneath.) The cache only works because fib is a pure function: the same argument always gives the same result. Never memoise a function that reads the clock, uses randomness or changes its arguments.
Common mistakes: forgetting return in the wrapper, so every decorated function returns None; writing @count_calls() with brackets when count_calls takes the function directly (brackets are for decorators that take their own settings, which are one level of closure deeper); leaving out functools.wraps; and caching functions whose arguments are lists, which functools.cache rejects because lists cannot be dictionary keys (pass a tuple).
Practice: How Often Is Each Case Asked For?, Commands That Sign Themselves Up, A Memory for Recurrences, with a Receipt
Context managers
A Statistics simulation should be reproducible, so it seeds the random generator. But the rest of the program may be using the same generator, and reseeding it changes everything that comes afterwards. A first version saves the generator's state and restores it by hand:
import random
random.seed(1)
state = random.getstate()
random.seed(42)
sample = [random.randint(1, 6) for _ in range(5)]
random.setstate(state)
print(sample)
print(random.random() == random.Random(1).random()) # the outer sequence is untouched
[6, 1, 1, 6, 3]
True
It works until the simulation raises an exception. Then setstate never runs and the generator stays reseeded for the rest of the program. Making it safe needs a try/finally around the simulation, every time. A context manager packages that "set up, then always clean up" pattern, and the with statement uses it. The easiest way to write one is contextlib.contextmanager: the code before yield runs on entering the with block, and the code after it (in a finally) runs on leaving, however the block ends:
import random
from contextlib import contextmanager
@contextmanager
def fixed_seed(seed):
state = random.getstate()
random.seed(seed)
try:
yield
finally:
random.setstate(state)
expected = random.Random(1).random() # the first value of the seed-1 sequence
random.seed(1)
with fixed_seed(42):
sample = [random.randint(1, 6) for _ in range(5)]
print(sample, random.random() == expected)
random.seed(1)
try:
with fixed_seed(7):
random.random()
raise RuntimeError("simulation failed")
except RuntimeError as e:
print("caught:", e)
print(random.random() == expected) # restored even though the block failed
[6, 1, 1, 6, 3] True
caught: simulation failed
True
The simulation code no longer mentions the generator's state at all, and the restoring cannot be forgotten or skipped.
A with block works with any object that follows the context-manager protocol from Level 4: __enter__ runs at the start (its return value is what as names) and __exit__ runs at the end, even after an exception. Files are the everyday example (with open(path) as f: closes the file for you), and the standard library has several more. contextlib.redirect_stdout sends everything printed inside the block to another file-like object, which is a neat way to capture a function's output in a test:
import io
from contextlib import redirect_stdout, suppress
buffer = io.StringIO()
with redirect_stdout(buffer):
print("hello from inside")
print(repr(buffer.getvalue()))
with suppress(KeyError): # "ignore this one error" in one line
{}["missing"]
print("carried on")
class Indent:
def __init__(self):
self.level = 0
def __enter__(self):
self.level += 1
return self
def __exit__(self, exc_type, exc, tb):
self.level -= 1
return False # do not swallow exceptions
def say(self, text):
print(" " * self.level + text)
ind = Indent()
ind.say("top")
with ind:
ind.say("nested")
with ind:
ind.say("deeper")
ind.say("top again")
'hello from inside\n'
carried on
top
nested
deeper
top again
Write a class with __enter__ and __exit__ when the object has other uses as well (like Indent); use @contextmanager for a one-off "before and after".
Common mistakes: forgetting the try/finally around yield in a @contextmanager function, so the clean-up is skipped when the block fails; returning True from __exit__, which silently swallows every exception raised in the block; and opening files without with, so they stay open if anything goes wrong.
Practice: A Simulation That Leaves No Trace, Keep Going, but Write It Down, Borrow the Settings, Then Give Them Back
Type hints
Here is a helper from a Machine Learning notebook. What does it accept, and what does it return?
def nearest(query, points, dist):
best = None
for p in points:
if best is None or dist(p, query) < dist(best, query):
best = p
return best
import math
print(nearest((5, 5), [(1, 1), (5, 4), (6, 6)], math.dist))
print(nearest((5, 5), [], math.dist))
(5, 4)
None
You have to read the whole body to learn that points is a list of points, that dist is a function of two points returning a number, and that the result may be None for an empty list. Type hints write that down in the signature: name: type for each parameter and -> type for the result.
import math
from typing import Callable
Point = tuple[float, float] # a type alias: a name for a type
def nearest(query: Point, points: list[Point],
dist: Callable[[Point, Point], float]) -> Point | None:
best: Point | None = None
for p in points:
if best is None or dist(p, query) < dist(best, query):
best = p
return best
print(nearest((5, 5), [(1, 1), (5, 4), (6, 6)], math.dist))
def word_lengths(words: list[str]) -> dict[str, int]:
return {w: len(w) for w in words}
print(word_lengths(["fig", "pear"]))
(5, 4)
{'fig': 3, 'pear': 4}
The shapes you need most are list[int], dict[str, float], tuple[int, int] (a pair) or tuple[int, ...] (any length), set[str], X | None for a value that may be missing, and Callable[[argument types], result type] for a function passed as an argument. A type alias such as Point gives a complicated type a readable name.
Hints are not checked when the program runs. Python ignores them, so a wrong call still runs until something breaks:
def double(x: int) -> int:
return x * 2
print(double("ab")) # no error: hints are not enforced at run time
abab
Their value comes from tools that read them without running anything: the editor on this site runs a type checker (mypy) in the background and underlines calls that do not match, and it uses the hints to offer better completions. They are also documentation that cannot drift out of date as easily as a comment, because the checker complains when the code and the hints disagree. The Point | None result is a good example: it reminds every caller to handle the empty case.
Common mistakes: expecting hints to reject bad arguments at run time (validate with an if and raise when that matters); hinting -> Point when the function can return None; writing list[float] for a parameter that also accepts tuples and generators (the more general Iterable[float] from collections.abc or typing says what you really need, which is the protocols idea again); and hinting so much of a short script that the hints are longer than the code.
Practice: Say What the Helpers Take, Where Does the Data Break Its Type?
Testing
You have written a median function for the Statistics path. A first version checks it by printing a result and looking at it:
def median(values):
s = sorted(values)
return s[len(s) // 2]
print(median([3, 1, 2]))
2
That looks right, so it is easy to move on. But one printed example checks one case, you have to remember what the right answer was, and the check disappears as soon as you delete the print. A test states the expected answer in code with assert, which does nothing when its condition is true and raises AssertionError when it is false. Grouping a few asserts into a function makes a check you can rerun after every change:
def median(values):
s = sorted(values)
return s[len(s) // 2]
def test_median():
assert median([3, 1, 2]) == 2
assert median([5]) == 5
assert median([4, 1, 3, 2]) == 2.5, "even length: mean of the middle two"
try:
test_median()
print("all passed")
except AssertionError as e:
print("FAILED:", e)
FAILED: even length: mean of the middle two
The test found the bug the eye missed: with an even number of values the median is the mean of the two middle ones. The message after the comma says which case failed. Good tests are mostly edge cases: empty input, one item, two items, even and odd lengths, duplicates, negative numbers, the largest allowed value. Write them before fixing the bug, so you know the fix works:
def median(values):
if not values:
raise ValueError("median of empty data")
s = sorted(values)
mid = len(s) // 2
return s[mid] if len(s) % 2 else (s[mid - 1] + s[mid]) / 2
def test_median():
assert median([3, 1, 2]) == 2
assert median([5]) == 5
assert median([4, 1, 3, 2]) == 2.5
assert median([-1, -1, 7]) == -1
try:
median([])
assert False, "empty data should raise"
except ValueError:
pass
test_median()
print("all passed")
all passed
When a trusted reference exists, a randomised test compares your function with it on hundreds of generated inputs. The seed keeps it reproducible, and small random inputs find edge cases you did not think of:
import random, statistics
def median(values):
s = sorted(values)
mid = len(s) // 2
return s[mid] if len(s) % 2 else (s[mid - 1] + s[mid]) / 2
rng = random.Random(0)
for _ in range(500):
data = [rng.randint(-5, 5) for _ in range(rng.randint(1, 8))]
assert median(data) == statistics.median(data), data
print("500 random cases agree")
500 random cases agree
The same idea checks a fast algorithm against a slow, obviously correct one, which is the most useful test in the Algorithms path. For floats, compare with math.isclose rather than ==. (Larger projects use a test runner such as pytest, which finds every function whose name starts with test_ and runs it; plain assert functions like these work with it unchanged.)
Common mistakes: testing only the example from the problem statement; comparing floats with ==; writing tests that print instead of assert, so a failure scrolls past unnoticed; and using assert to validate user input in real code (asserts can be switched off with python -O; raise a proper exception instead).
Practice: A Test That Catches Every Broken Median
Debugging
A sliding-window function from the Algorithms path should return the largest sum of k consecutive numbers. It passes the example you tried, but a hidden test fails:
def max_window_sum(nums, k):
window = sum(nums[:k])
best = window
for i in range(k, len(nums) - 1):
window += nums[i] - nums[i - k]
best = max(best, window)
return best
print(max_window_sum([1, 3, 2, 6, 1, 1], 2))
8
The tempting first move is to stare at the code and change things until the test passes. That is slow, and it often "fixes" the symptom while leaving the cause. Debugging goes much faster as a method: reproduce the failure, shrink it to the smallest input that still fails, look at what the program really does on that input, and only then fix it and keep the failing case as a test.
To reproduce and shrink at once, compare with a slow version that is obviously right, on every small input in turn, and stop at the first disagreement. itertools.product generates all the short lists:
from itertools import product
def max_window_sum(nums, k):
window = sum(nums[:k])
best = window
for i in range(k, len(nums) - 1):
window += nums[i] - nums[i - k]
best = max(best, window)
return best
def brute(nums, k):
return max(sum(nums[i:i + k]) for i in range(len(nums) - k + 1))
def first_failure():
for n in range(1, 5):
for k in range(1, n + 1):
for nums in product([0, 1], repeat=n):
got, want = max_window_sum(list(nums), k), brute(nums, k)
if got != want:
return nums, k, got, want
print(first_failure())
((0, 1), 1, 0, 1)
Two numbers and a window of one is small enough to follow by hand. To look inside, print the state at each step. The = inside an f-string prints the expression and its value, which makes quick debugging output easy to read:
def max_window_sum(nums, k):
window = sum(nums[:k])
best = window
print(f"start: {window=} {best=}")
for i in range(k, len(nums) - 1):
window += nums[i] - nums[i - k]
best = max(best, window)
print(f"{i=} {window=} {best=}")
return best
print(max_window_sum([0, 1], 1))
print(list(range(1, len([0, 1]) - 1)))
start: window=0 best=0
0
[]
The loop body never runs: range(k, len(nums) - 1) stops one index early, so the last window is never considered. The fix is range(k, len(nums)), and max_window_sum([0, 1], 1) == 1 becomes a permanent test.
A few more habits pay off constantly. Read an error message from the bottom: the last line names the exception and its message, and the line above it shows where it happened. Check your assumptions with assert in the middle of the code (assert lo <= hi, (lo, hi)), so that a wrong state fails loudly where it starts rather than much later. And when you are stuck, explain the code line by line to someone, or to a rubber duck; saying what each line should do is often enough to spot the one that does not. Printing and asserting work everywhere, including in the browser on this site; on your own computer, breakpoint() also opens Python's step-by-step debugger.
Common mistakes: changing several things at once, so you do not know which change mattered; debugging a large input instead of shrinking it first; removing the debugging prints but not adding the failing case as a test; and assuming the bug is in the part you were not sure about. Often it is in the part you were sure was fine.
Practice: Every Row Books the Same Seat, Four Bug Reports for One Short Function, The Shortest List That Tells Them Apart
The standard library toolbox
Four modules solve a surprising share of everyday problems: itertools (ways to loop), functools (ways to combine functions), heapq (the heap) and bisect (binary search on a sorted list). The best way to learn them is to notice the hand-written loop each one replaces.
Every pair. The Algorithms path's "count the pairs" problem uses two nested index loops. A first version:
people = ["ana", "ben", "chen", "dev"]
pairs = []
for i in range(len(people)):
for j in range(i + 1, len(people)):
pairs.append((people[i], people[j]))
print(len(pairs), pairs[:3])
6 [('ana', 'ben'), ('ana', 'chen'), ('ana', 'dev')]
itertools.combinations produces exactly those pairs (or triples, or any size), with no index arithmetic to get wrong. Its relatives cover the other common shapes: permutations for orderings, product for every combination of choices (the grid of hyperparameters in Machine Learning), and accumulate for running totals, which are the prefix sums of the Algorithms path:
from itertools import combinations, permutations, product, accumulate
people = ["ana", "ben", "chen", "dev"]
print(len(list(combinations(people, 2))), len(list(permutations("abc"))))
print(list(product([0.1, 0.01], [16, 32])))
print(list(accumulate([3, 1, 4, 1, 5])))
6 6
[(0.1, 16), (0.1, 32), (0.01, 16), (0.01, 32)]
[3, 4, 8, 9, 14]
The k largest. To find the three highest scores, sorted(scores, reverse=True)[:3] sorts everything, O(n log n), to keep three. heapq.nlargest(k, ...) keeps a heap of only k items as it goes, O(n log k), and takes a key like sorted. The heap functions themselves (heappush, heappop) are the priority queue of Dijkstra's algorithm, as you saw in Level 4:
import heapq
scores = {"ana": 88, "ben": 95, "chen": 72, "dev": 91, "eli": 60}
print(heapq.nlargest(3, scores, key=scores.get))
print(heapq.nsmallest(2, scores.values()))
['ben', 'dev', 'ana']
[60, 72]
Looking up a band. Level 1 graded a score with an elif chain. With many bands, keep the boundaries in a sorted list and let bisect find the right one by binary search. bisect_right(cuts, x) returns how many boundaries are at or below x, which is the index of the band. The same call finds a value's position among sorted data, which is how a percentile rank is computed in Statistics:
from bisect import bisect_right, insort
cuts = [60, 70, 80, 90]
grades = "FDCBA"
print([grades[bisect_right(cuts, s)] for s in [55, 60, 79, 90, 100]])
data = [2, 3, 5, 8, 13]
insort(data, 7) # insert, keeping the list sorted
print(data, bisect_right(data, 6) / len(data)) # share of values at or below 6
['F', 'D', 'C', 'A', 'A']
[2, 3, 5, 7, 8, 13] 0.5
Fixing an argument and folding a list. functools.partial makes a new function with some arguments filled in, which is often clearer than a lambda. functools.reduce combines a list into one value with a two-argument function; use it when no built-in (sum, max, math.prod) already does the job:
from functools import partial, reduce
import math
round2 = partial(round, ndigits=2)
print(list(map(round2, [3.14159, 2.71828])))
print(reduce(math.gcd, [84, 126, 210]))
[3.14, 2.72]
42
Common mistakes: forgetting that itertools functions return one-use iterators; generating permutations of more than about ten items (10! is already 3.6 million); using bisect on a list that is not sorted, which gives wrong answers without an error; and using reduce where sum, max or a plain loop would be clearer.
Practice: Which Chapter Is This Bookmark In?
Designing clean function interfaces
A feature-scaling helper for the Machine Learning path grew over time. A first version has a flag for every new need:
def scale(v, flag, inplace):
if flag:
lo, hi = min(v), max(v)
out = [(x - lo) / (hi - lo) for x in v]
else:
m = sum(v) / len(v)
s = (sum((x - m) ** 2 for x in v) / len(v)) ** 0.5
out = [(x - m) / s for x in v]
if inplace:
v[:] = out
else:
return out
data = [2.0, 4.0, 6.0]
print(scale(data, True, False))
print(scale(data, False, True), data)
[0.0, 0.5, 1.0]
None [-1.224744871391589, 0.0, 1.224744871391589]
Each problem is small, but together they make the function easy to misuse. scale(data, True, False) says nothing at the call site: which flag is which? The return type depends on an argument (a list, or None), so result = scale(data, False, True) quietly gives None. And the in-place option changes the caller's data, which is the aliasing surprise from Level 1 in a new form. A few rules make an interface hard to call wrongly:
- Name the choices. Use a keyword argument with meaningful values (
method="minmax") instead of a boolean, and make it keyword-only by putting a*before it, so callers must write the name. - Return one kind of thing. A function either returns a new value or changes its argument and returns
None(aslist.sortdoes), never both depending on a flag. - Do not change the inputs unless that is the function's whole purpose and its name says so.
- Fail loudly on bad input, with a message that says what was expected.
- Do one thing. When two branches share no code, they are two functions; a small dictionary can map names to them.
def minmax(v):
lo, hi = min(v), max(v)
if hi == lo:
raise ValueError("cannot min-max scale constant data")
return [(x - lo) / (hi - lo) for x in v]
def zscore(v):
m = sum(v) / len(v)
s = (sum((x - m) ** 2 for x in v) / len(v)) ** 0.5
return [(x - m) / s for x in v]
SCALERS = {"minmax": minmax, "zscore": zscore}
def scale(v, *, method="zscore"):
"""Return a scaled copy of v. method is 'minmax' or 'zscore'."""
if method not in SCALERS:
raise ValueError(f"unknown method {method!r}; choose from {sorted(SCALERS)}")
return SCALERS[method](list(v))
data = [2.0, 4.0, 6.0]
print(scale(data, method="minmax"), data)
print([round(x, 3) for x in scale(data)])
for bad in (lambda: scale(data, "minmax"), lambda: scale(data, method="max")):
try:
bad()
except (TypeError, ValueError) as e:
print(type(e).__name__)
[0.0, 0.5, 1.0] [2.0, 4.0, 6.0]
[-1.225, 0.0, 1.225]
TypeError
ValueError
Every call now reads as a sentence (scale(data, method="minmax")), passing the method positionally is refused, an unknown method gets a helpful message, and data is never changed. The two scalers can also be tested and reused on their own. A good interface is one where the obvious way to call the function is the correct one.
Common mistakes: boolean flags that switch between unrelated behaviours; returning -1, None or False to mean "error" (callers forget to check; raise instead); long lists of positional parameters that callers must get in the right order; and functions that both compute and print, which makes them impossible to reuse or test.
Practice: A Sampler You Cannot Call Wrongly, Every Combination of Settings, Best First, Checked at the Door
Practice
These problems are required on this path at this level; the path adds one problem from each of the level's topics, and one for each construct the others do not use, picked for you and kept.
- Say What the Helpers Take
- Every Row Books the Same Seat
- A Test That Catches Every Broken Median
- How Often Is Each Case Asked For?
- Commands That Sign Themselves Up
- A Memory for Recurrences, with a Receipt
Checklist
- Write a decorator with a closure and
functools.wraps, and memoise a pure recursive function withfunctools.cache. - Guarantee clean-up with
with, write a context manager with@contextmanagerandtry/finally, and useredirect_stdoutandsuppress. - Add type hints to a function, including
X | None,Callableand a type alias, and explain why they are not checked at run time. - Write test functions with
assert, cover edge cases, and compare against a reference on seeded random inputs. - Debug by reproducing, shrinking to a minimal failing input, printing state with
f"{x=}", then fixing and keeping the test. - Replace nested pair loops, top-k sorting,
elifbands and running totals withitertools,heapq,bisectandfunctools. - Design functions with keyword-only options, a single return type, no hidden changes to their inputs and clear errors.