Level 3 · Python Fluency: Python Computation
Levels 1 and 2 were about writing individual pieces of code well. Level 3 is about how data flows through a program: where values come from, how they are passed from one step to the next, and what happens when one of them is wrong. The central idea is the iterator, the object behind every for loop, and its easy-to-write form, the generator. With them you can process a stream of a billion readings using the memory of a single one, and build a calculation as a pipeline of small, separate steps.
The second idea is that functions are values. You have already passed len as a sort key; here you will pass loss functions, build functions that remember settings (closures) and write code that works with any function you hand it. The last sections deal with the real world's untidiness: exceptions for data that is not what you expected, modules so you do not reinvent the standard library, reading text data properly, and collections.Counter and defaultdict for the counting and grouping that every data task needs.
This level is where the paths start to lean on the language. The Statistics path reads data from text and computes summaries in a single pass; the Machine Learning path passes loss functions and scalers around as values; the Algorithms path uses generators to walk trees and Counter to compare anagrams. Each section points out where.
Iterators
A table arrives as a list of rows whose first row is a header. A first version slices the header off:
rows = [["name", "score"], ["ana", 7], ["ben", 9], ["chen", 8]]
header = rows[0]
total = 0
for row in rows[1:]:
total += row[1]
print(header, total)
['name', 'score'] 24
rows[1:] copies every row except the first just to skip one item, and it only works because rows is a list. If the data came from a file or a network connection, there would be nothing to slice. What for really does is ask an object for an iterator with iter(...), then call next(...) on it until it runs out. You can do those steps yourself:
rows = [["name", "score"], ["ana", 7], ["ben", 9], ["chen", 8]]
it = iter(rows)
header = next(it) # take one item by hand
total = 0
for row in it: # the loop carries on from where next() stopped
total += row[1]
print(header, total)
['name', 'score'] 24
Nothing is copied: the iterator is a bookmark into the data that moves forward one item at a time. That is also why an iterator can only be used once. Once it reaches the end it stays there, and a second loop over it sees nothing:
pairs = zip([1, 2, 3], "abc")
print(list(pairs))
print(list(pairs)) # already used up
it = iter([10, 20])
print(next(it), next(it), next(it, "done"))
[(1, 'a'), (2, 'b'), (3, 'c')]
[]
10 20 done
The distinction to keep in mind is between an iterable (something you can loop over as often as you like, such as a list, string, dictionary or range) and an iterator (a one-pass stream, such as the result of iter, zip, map, enumerate or a file). next(it, default) returns the default instead of raising StopIteration at the end, which gives a tidy way to find the first match:
readings = [12, 15, 31, 18, 40]
print(next((r for r in readings if r > 30), None))
print(next((r for r in readings if r > 99), None))
31
None
Common mistakes: looping twice over a zip, map or file and getting nothing the second time (turn it into a list if you need it twice); calling next on a list (next([1, 2]) raises TypeError; call iter first); and calling len on an iterator, which does not know how many items remain.
Practice: The Header Comes First, Three High Readings in a Row
Generators and yield
A sensor log has one reading per line, with blank lines and # comments mixed in, and you want the mean. A first version collects every reading into a list and then averages it:
import io
log = io.StringIO("# sensor 7\n20.5\n21.0\n\n19.5\n# restart\n22.0\n")
def read_values(f):
values = []
for line in f:
line = line.strip()
if line and not line.startswith("#"):
values.append(float(line))
return values
values = read_values(log)
print(sum(values) / len(values))
20.75
(io.StringIO makes a string behave like an open file, which is how the examples on this site stand in for real files.) The list is the problem. A day of readings at 100 per second is over eight million floats held in memory at once, only to be added up and thrown away. A generator function uses yield instead of append: each time the caller asks for the next value, the function runs until its next yield, hands over that one value and pauses there.
import io
log = io.StringIO("# sensor 7\n20.5\n21.0\n\n19.5\n# restart\n22.0\n")
def read_values(f):
for line in f:
line = line.strip()
if line and not line.startswith("#"):
yield float(line)
def mean(values):
n = total = 0
for x in values:
n += 1
total += x
return total / n
print(mean(read_values(log)))
20.75
Only one line is ever in memory. The two functions also stay separate: read_values knows about the file format, mean knows about arithmetic, and neither knows about the other. You can chain as many steps as you like into a pipeline, and nothing runs until the last step asks for a value.
A generator can also yield a running result instead of only a final one. This version of the mean updates itself with each value (the Statistics path derives the formula mean += (x - mean) / n, which avoids adding up a huge total), and runs over a million values without ever building a list:
import sys
def running_mean(values):
n, m = 0, 0.0
for x in values:
n += 1
m += (x - m) / n
yield m
stream = (i % 10 for i in range(1_000_000)) # a generator expression
for m in running_mean(stream):
pass
print(round(m, 6))
print(sys.getsizeof(i % 10 for i in range(1_000_000)) < 500)
print(sys.getsizeof([i % 10 for i in range(1_000_000)]) > 1_000_000)
4.5
True
True
The generator object is a few hundred bytes however many values it will produce; the list of the same million values takes megabytes. Generators can even be infinite, because they only compute what is asked for. itertools.islice takes the first few:
from itertools import islice
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
print(list(islice(fibonacci(), 10)))
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
Common mistakes: calling a generator function and expecting it to run (read_values(log) does nothing until something loops over it); using the same generator twice (like every iterator, it is used up); writing return values in a generator when you meant yield; and converting a generator to a list "to be safe", which throws away the memory saving.
Practice: Waiting Longer After Each Failure, Mean and Spread While the Data Arrives
Functions as values
The Machine Learning path measures how wrong a model's predictions are with a loss. Two common ones are the mean squared error and the mean absolute error. A first version writes each as its own function:
def mean_squared_error(pred, actual):
total = 0
for p, a in zip(pred, actual):
total += (p - a) ** 2
return total / len(pred)
def mean_absolute_error(pred, actual):
total = 0
for p, a in zip(pred, actual):
total += abs(p - a)
return total / len(pred)
pred, actual = [4.0, 1.5, 3.0], [3.0, 2.0, 3.0]
print(mean_squared_error(pred, actual), mean_absolute_error(pred, actual))
0.4166666666666667 0.5
In maths notation they compute \frac{1}{n}\sum_{i=1}^{n}(p_i - a_i)^2 and \frac{1}{n}\sum_{i=1}^{n}\lvert p_i - a_i\rvert, where p_i is the i-th prediction and a_i the actual value. The two functions are identical except for one expression, the part after the \sum. In Python a function is a value like a number or a list: you can store it in a variable, put it in a dictionary and pass it to another function. So write the shared loop once and pass in the part that differs:
def mean_loss(pred, actual, loss):
return sum(loss(p, a) for p, a in zip(pred, actual)) / len(pred)
def squared(p, a):
return (p - a) ** 2
pred, actual = [4.0, 1.5, 3.0], [3.0, 2.0, 3.0]
print(mean_loss(pred, actual, squared))
print(mean_loss(pred, actual, lambda p, a: abs(p - a)))
0.4166666666666667
0.5
A function that takes or returns another function is called a higher-order function. You have already used several: sorted, min and max take a key, and map and filter take a function to apply. A lambda is a function written as a single expression, with no name and no return; it is meant for short functions used once, like the absolute error above. Anything longer deserves a def and a name.
Storing functions in a dictionary replaces a long if/elif chain that picks behaviour by name:
import operator
ops = {"+": operator.add, "-": operator.sub, "*": operator.mul}
def evaluate(a, op, b):
return ops[op](a, b)
print(evaluate(6, "*", 7), evaluate(10, "-", 4))
losses = {"mse": squared, "mae": lambda p, a: abs(p - a)}
print(sorted(losses))
42 6
['mae', 'mse']
Common mistakes: calling the function when you meant to pass it (mean_loss(pred, actual, squared(1, 2)) passes a number, not a function); assigning a lambda to a name (f = lambda x: x + 1; write def f(x):, which gives the function a proper name in error messages); and writing lambdas so long that the reader has to stop and decode them.
Practice: One Number to Bake Every Day
Closures
Feature scaling in the Machine Learning path subtracts the training data's mean and divides by its standard deviation, and it must use the training values for every later input. A first version passes those numbers along with every call:
def scale(x, mean, std):
return (x - mean) / std
train = [10, 20, 30, 40]
m = sum(train) / len(train)
s = (sum((x - m) ** 2 for x in train) / len(train)) ** 0.5
print([round(scale(x, m, s), 3) for x in train])
print(round(scale(35, m, s), 3))
[-1.342, -0.447, 0.447, 1.342]
0.894
Every caller must carry m and s around and pass them in the right order; swap them and the results are quietly wrong. A closure is a function defined inside another function that remembers the variables around it, even after the outer function has returned. The outer function computes the settings once and returns a scaler that already knows them:
def make_scaler(train):
m = sum(train) / len(train)
s = (sum((x - m) ** 2 for x in train) / len(train)) ** 0.5
def scale(x):
return (x - m) / s
return scale
scale = make_scaler([10, 20, 30, 40])
print([round(scale(x), 3) for x in [10, 20, 30, 40]])
print(round(scale(35), 3))
[-1.342, -0.447, 0.447, 1.342]
0.894
The caller cannot pass the wrong mean because it never sees the mean. A closure that needs to change a remembered variable must declare it nonlocal; without that, assigning to it would create a new local variable instead:
def make_counter():
count = 0
def tick():
nonlocal count
count += 1
return count
return tick
tick = make_counter()
tick(); tick()
print(tick())
3
Closures remember variables, not values, and that produces one famous surprise. Functions created in a loop all see the loop variable's final value, unless you capture the current value with a default argument:
late = [lambda x: x * k for k in range(3)]
print([f(10) for f in late])
early = [lambda x, k=k: x * k for k in range(3)]
print([f(10) for f in early])
[20, 20, 20]
[0, 10, 20]
When a closure starts to hold several settings and needs several operations, it is time for a class, which is where Level 4 begins.
Common mistakes: forgetting nonlocal and getting UnboundLocalError; creating functions in a loop and expecting each to remember its own loop value; and returning scale() (the result of calling it) instead of scale (the function).
Practice: A Thermostat for Every Room, A Scaler That Remembers Its Training Data
Exceptions
A survey column should hold numbers, but real data contains blanks, typos and the odd n/a. A first version checks each value before converting it:
raw = ["12", "7", "n/a", "-3", "2.5", ""]
values = []
for s in raw:
if s.isdigit():
values.append(int(s))
print(values)
[12, 7]
The check is wrong in both directions: "-3" and "2.5" are perfectly good numbers, but isdigit rejects them. Predicting everything that could go wrong is hard, and the rules usually end up duplicating the conversion itself. Python's style is the opposite: try the operation, and handle the failure if it happens. float already knows exactly what a valid number looks like and raises ValueError when it sees something else:
raw = ["12", "7", "n/a", "-3", "2.5", ""]
values, bad = [], []
for s in raw:
try:
values.append(float(s))
except ValueError:
bad.append(s)
print(values)
print(bad)
[12.0, 7.0, -3.0, 2.5]
['n/a', '']
The try block holds the code that might fail, and the except clause names the error it handles. Name the specific exception: a bare except: also catches typos in your own code (NameError) and hides them.
Your own functions should raise exceptions too, when they are asked to do something impossible. Returning 0 or None for the mean of an empty list lets the mistake travel on and cause a confusing failure much later; raising at once, with a clear message, stops it where it started:
def mean(values):
if not values:
raise ValueError("mean of an empty list")
return sum(values) / len(values)
try:
print(mean([]))
except ValueError as e:
print("error:", e)
finally:
print("finally always runs")
print(mean([1, 2, 6]))
error: mean of an empty list
finally always runs
3.0
except ... as e gives you the exception object, whose text is the message. A finally block runs whether or not an error happened, which is where clean-up code goes (Level 5's context managers do this more neatly). The standard library follows the same rule: statistics.mean([]) raises statistics.StatisticsError.
Common mistakes: a bare except: or except Exception: around a large block, which turns every bug into silence; putting too much inside try, so you cannot tell which line failed; catching an error and returning a made-up value such as -1; and using exceptions for ordinary control flow where a simple if is clearer (if key in d is fine).
Practice: Sharing the Prize Money, or Refusing To, Try Every Parser in Turn
Modules and imports
The Statistics path needs the number of ways to choose 3 items from 10. A first version writes its own factorial:
def factorial(n):
result = 1
for k in range(2, n + 1):
result *= k
return result
def choose(n, k):
return factorial(n) // (factorial(k) * factorial(n - k))
print(choose(10, 3))
120
It is correct, but it is also already written, tested and faster in the standard library. A module is a file of Python code with its own names, and import makes those names available. Python comes with hundreds of modules; knowing roughly what they contain saves you from writing and debugging code that exists:
import math
import statistics as st
from collections import deque
print(math.comb(10, 3), math.factorial(5), math.prod([2, 3, 4]))
print(math.dist((0, 0), (3, 4)), math.isqrt(50))
print(st.median([3, 1, 4, 1, 5]))
q = deque([1, 2, 3])
q.appendleft(0)
print(list(q))
120 120 24
5.0 7
3
[0, 1, 2, 3]
The three forms each have a use. import math keeps the module name in front (math.comb), so a reader always knows where a function comes from and your own log cannot clash with math.log. import statistics as st gives a long name a short alias. from collections import deque brings in one name you use often. Avoid from math import *, which dumps every name into your program and makes it impossible to tell which names are yours.
Every module has a __name__. When a file is run directly its __name__ is "__main__", and when it is imported it is the module's name. That is why scripts often end with a guard, so that importing the file for its functions does not also run its demonstration code:
import math
print(math.__name__, __name__)
def main():
print("running as a script")
if __name__ == "__main__":
main()
math __main__
running as a script
On this site your code runs in the browser with the standard library only, which is a large toolbox: math, statistics, random, collections, itertools, functools, heapq, bisect, json, re and more.
Common mistakes: naming your own file or variable after a module (random = 5 and then random.random() fails); from module import *; importing inside a function that runs a million times (importing again is cheap but not free; import at the top); and reinventing math.comb, math.dist or statistics.median instead of using them, at least to check your own versions.
Practice: A Summary Line from the Standard Library, A Deadline in Working Days
Reading text data
Data usually arrives as text: a CSV export, a pasted table, a log. Here is a small CSV file of cities, and you want the mean population. A first version splits each line on commas:
text = """city,country,population
Lagos,Nigeria,15400000
"Washington, D.C.",USA,690000
Oslo,Norway,709000
"""
lines = text.strip().split("\n")
for line in lines[1:]:
print(line.split(","))
['Lagos', 'Nigeria', '15400000']
['"Washington', ' D.C."', 'USA', '690000']
['Oslo', 'Norway', '709000']
The second row breaks: CSV allows a field to contain a comma if the field is in quotes, and split(",") knows nothing about quotes. Now the population is in the fourth column for that row only, and converting column 3 would crash or, worse, read the wrong value. The csv module implements the full format, and it reads from any iterator of lines, including an io.StringIO:
import csv, io
text = """city,country,population
Lagos,Nigeria,15400000
"Washington, D.C.",USA,690000
Oslo,Norway,709000
"""
reader = csv.DictReader(io.StringIO(text))
rows = list(reader)
print(rows[1]["city"], rows[1]["population"])
pops = [int(r["population"]) for r in rows]
print(sum(pops) / len(pops))
Washington, D.C. 690000
5599666.666666667
csv.DictReader uses the first row as the header and turns every later row into a dictionary, so the code says r["population"] instead of r[2] and keeps working if someone adds a column. All values arrive as strings; converting them is your job. csv.reader does the same without the header handling, giving each row as a list.
For simpler text, a few string methods do most of the work. str.splitlines() splits on line endings of every kind (including the Windows \r\n) and does not leave an empty string at the end the way split("\n") does:
text = "3 4\r\n5 12\r\n"
print(text.split("\n"))
print(text.splitlines())
for line in text.splitlines():
a, b = map(int, line.split())
print(a, b, (a * a + b * b) ** 0.5)
['3 4\r', '5 12\r', '']
['3 4', '5 12']
3 4 5.0
5 12 13.0
Reading line by line from a file object (for line in f:) is also an iterator, so a large file never has to fit in memory. That is the pattern from the generators section: a generator that reads and cleans lines, feeding a calculation that reads one value at a time.
Common mistakes: splitting CSV with split(","); forgetting that every field arrives as a string ("10" > "9" is False because strings compare letter by letter); leaving "\n" or "\r" on the end of the last field (use strip() or splitlines()); and reading an entire large file with .read() when a loop over its lines would do.
Practice: A Settings File with Continued Lines, A Three-Stage Log Pipeline
Counting and grouping with collections
Level 1 counted words with a dictionary and get. Grouping is the same shape: here, a first version groups words by their first letter, checking each time whether the key exists yet:
words = ["apple", "bean", "avocado", "cherry", "beet", "apricot"]
groups = {}
for w in words:
if w[0] not in groups:
groups[w[0]] = []
groups[w[0]].append(w)
print(groups)
{'a': ['apple', 'avocado', 'apricot'], 'b': ['bean', 'beet'], 'c': ['cherry']}
The "does the key exist yet?" test is pure bookkeeping. collections.defaultdict takes a function that makes the starting value, and calls it automatically the first time a missing key is used. For counting, collections.Counter goes further: give it any iterable and it counts everything in one step:
from collections import Counter, defaultdict
words = ["apple", "bean", "avocado", "cherry", "beet", "apricot"]
groups = defaultdict(list)
for w in words:
groups[w[0]].append(w)
print(dict(groups))
text = "the cat saw the dog and the cat ran"
counts = Counter(text.split())
print(counts.most_common(2))
print(counts["the"], counts["bird"])
{'a': ['apple', 'avocado', 'apricot'], 'b': ['bean', 'beet'], 'c': ['cherry']}
[('the', 3), ('cat', 2)]
3 0
A Counter returns 0 for a missing key instead of raising KeyError, and most_common(k) gives the k largest counts, which is the mode of the Statistics path in one call. Counters can also be compared and combined, which the Algorithms path uses to test whether two words are anagrams:
from collections import Counter
print(Counter("below") == Counter("elbow"))
a = Counter(red=3, blue=1)
b = Counter(red=1, green=2)
print(a + b)
print(a - b)
True
Counter({'red': 4, 'green': 2, 'blue': 1})
Counter({'red': 2, 'blue': 1})
Both classes are still dictionaries underneath, so everything from Level 1 (items(), in, looping over keys) works on them.
Common mistakes: reading a missing key of a defaultdict just to check it (that creates the key; use in to test); printing a defaultdict and being puzzled by its long repr (wrap it in dict(...)); and building a Counter by hand in a loop when Counter(items) does it in one call.
Practice: Rainfall Totals from a Spreadsheet Export, Customers Who Bought This Also Bought
Practice
These problems are required on this path at this level; the path adds one problem from each of the level's topics, and one for each construct the others do not use, picked for you and kept.
- One Number to Bake Every Day
- Sharing the Prize Money, or Refusing To
- A Scaler That Remembers Its Training Data
- The Header Comes First
- Three High Readings in a Row
- Mean and Spread While the Data Arrives
Checklist
- Explain the difference between an iterable and an iterator, and use
iterandnext(with a default) directly. - Write a generator with
yield, chain generators into a pipeline and compute a running mean in constant memory. - Pass functions as arguments, store them in dictionaries and keep
lambdafor short, one-off functions. - Build a closure that remembers settings, use
nonlocal, and avoid the late-binding surprise in loops. - Try an operation and handle the specific exception it raises, and raise
ValueErrorwith a clear message from your own functions. - Import modules in the three common forms, protect demonstration code with
if __name__ == "__main__":, and reach formath,statisticsandcollectionsbefore writing your own. - Read CSV with
csv.DictReaderoverio.StringIO, split lines withsplitlines, and convert fields from strings. - Count with
Counterand group withdefaultdict(list).