Level 4 · Python Fluency: Python Abstraction

Level 3 ended with a closure that remembered a mean and a standard deviation, and a warning: when remembered settings multiply and need several operations, you want a class. This level is about abstraction, packaging data together with the operations that make sense on it, so that the rest of the program can use the package without knowing how it works inside. A priority queue that only lets you push and pop, a vector that adds with +, a model with fit and predict: each hides its details behind a small, clear interface.

The level moves from the basics of classes (state and behaviour) to the choice every design faces, composition or inheritance, and then to the tools that cut the boilerplate: @dataclass writes the routine methods for you, @property turns a computed value into something that looks like a plain attribute, and special methods such as __repr__, __eq__, __lt__, __len__ and __iter__ let your objects work with print, ==, sorted, len and for. The last section explains the idea that ties it together: in Python, what an object can do matters more than what class it belongs to.

These are the building blocks of the other paths' larger programs. The Algorithms path wraps a heap in a priority queue and a grid in a graph; the Statistics path keeps running summaries of a stream; the Machine Learning path gives every model the same fit/predict interface so that they can be swapped and compared.

Classes and objects

You want the count, mean and variance of a stream of values, updated one value at a time so that the stream never has to be stored. A first version keeps the state in a tuple and passes it in and out of a function:

def update(state, x):
    n, mean, m2 = state
    n += 1
    delta = x - mean
    mean += delta / n
    m2 += delta * (x - mean)
    return n, mean, m2

state = (0, 0.0, 0.0)
for x in [4, 8, 6, 9, 3, 6, 7, 5]:
    state = update(state, x)
n, mean, m2 = state
print(n, mean, m2 / n)

8 6.0 3.5

(The update rule is Welford's method, which the Statistics path uses for a variance in one pass.) The tuple works, but nothing says what its three parts mean, the caller must remember to reassign state every time, and anyone can build a nonsensical state by hand. A class bundles the data (the object's attributes) with the functions that use it (its methods):

class RunningStats:
    """Count, mean and variance of a stream, in one pass."""

    def __init__(self):
        self.n = 0
        self.mean = 0.0
        self._m2 = 0.0

    def add(self, x):
        self.n += 1
        delta = x - self.mean
        self.mean += delta / self.n
        self._m2 += delta * (x - self.mean)

    def variance(self):
        return self._m2 / self.n

stats = RunningStats()
for x in [4, 8, 6, 9, 3, 6, 7, 5]:
    stats.add(x)
print(stats.n, stats.mean, stats.variance())

other = RunningStats()
other.add(10)
print(other.n, stats.n)

8 6.0 3.5
1 8

class RunningStats: defines a new type, and RunningStats() creates an object (an instance) of it, calling __init__ to set up its attributes. Inside a method, self is the object the method was called on: stats.add(x) runs add with self set to stats. Each object has its own attributes, so other and stats do not interfere.

The leading underscore in _m2 is a convention meaning "internal: not part of the interface". Python does not enforce it, but it tells readers which attributes they may rely on (n, mean) and which may change without notice.

Common mistakes: forgetting self as the first parameter of a method (the call then fails with a TypeError about the number of arguments); writing mean = 0.0 instead of self.mean = 0.0 in __init__, which sets a local variable that vanishes when the method returns, so the object never gets a mean attribute; and putting a list at class level (class Basket: items = []), which is shared by every object of the class, the same trap as a mutable default argument.

Practice: Accounts With Their Own History, A Fleet of Floor Robots

Composition

Dijkstra's algorithm in the Algorithms path needs a priority queue: push items with priorities, and always pop the one with the smallest priority. The heapq module keeps a plain list in heap order. A first version makes the queue a kind of list by inheriting from list:

import heapq

class PriorityQueue(list):
    def push(self, priority, item):
        heapq.heappush(self, (priority, item))
    def pop_min(self):
        return heapq.heappop(self)

pq = PriorityQueue()
pq.push(5, "e")
pq.push(1, "a")
pq.push(3, "c")
pq.append((9, "z"))        # a list method: allowed, but breaks the heap order
pq.insert(0, (7, "q"))     # so is this
print(pq.pop_min())

(7, 'q')

The queue returned priority 7 before priority 1. Because it is a list, it has every list method (append, insert, sort, item assignment), and any of them can silently break the heap order that heapq depends on. Composition builds the queue out of a list instead: the list becomes a private attribute, and the class offers only the operations that keep it valid.

import heapq

class PriorityQueue:
    def __init__(self):
        self._heap = []
    def push(self, priority, item):
        heapq.heappush(self._heap, (priority, item))
    def pop_min(self):
        return heapq.heappop(self._heap)
    def peek(self):
        return self._heap[0]
    def __len__(self):
        return len(self._heap)

pq = PriorityQueue()
for p, item in [(5, "e"), (1, "a"), (3, "c")]:
    pq.push(p, item)
print(pq.peek(), len(pq))
print(pq.pop_min(), pq.pop_min(), len(pq))

(1, 'a') 3
(1, 'a') (3, 'c') 1

Now the only way to change the queue is through push and pop_min, so the heap order cannot be broken from outside. The interface is also yours to keep: you could swap the heap for a sorted list or a deque later, and no caller would notice. (__len__ makes len(pq) work; the special-methods section below explains how.)

Composition is the relationship "has a": a priority queue has a list, a k-nearest-neighbours model has a scaler and a list of training points, a simulation has a random.Random generator. It is the default way to build larger objects from smaller ones.

Common mistakes: inheriting from list or dict to save a few lines, and inheriting all their methods with it; reaching into another object's internals (pq._heap.append(...)) instead of asking for the operation you need; and wrapping so many layers that a simple call passes through five objects. One or two is usually plenty.

Practice: Dice That Keep Their Own Luck, A Bank of Lifts

Inheritance

The Machine Learning path compares simple classifiers. Each one learns from examples with fit, labels new inputs with predict, and is judged by its accuracy. A first version writes the accuracy method into every class:

class MajorityClass:
    def fit(self, xs, ys):
        self.label = max(set(ys), key=ys.count)
    def predict(self, x):
        return self.label
    def accuracy(self, xs, ys):
        return sum(self.predict(x) == y for x, y in zip(xs, ys)) / len(ys)

class Threshold:
    def __init__(self, cut):
        self.cut = cut
    def fit(self, xs, ys):
        pass
    def predict(self, x):
        return 1 if x >= self.cut else 0
    def accuracy(self, xs, ys):
        return sum(self.predict(x) == y for x, y in zip(xs, ys)) / len(ys)

xs, ys = [1, 2, 3, 6, 7, 8], [0, 0, 1, 1, 1, 1]
for model in (MajorityClass(), Threshold(3)):
    model.fit(xs, ys)
    print(type(model).__name__, model.accuracy(xs, ys))

MajorityClass 0.6666666666666666
Threshold 1.0

The two accuracy methods are copies, and the next five models would make seven. Inheritance puts the shared behaviour in a base class; each subclass inherits everything and overrides only what differs. super() calls the base class's version of a method, which is how a subclass extends __init__ rather than replacing it:

class Classifier:
    def __init__(self, name):
        self.name = name
    def fit(self, xs, ys):
        pass                                   # some models have nothing to learn
    def predict(self, x):
        raise NotImplementedError
    def accuracy(self, xs, ys):
        return sum(self.predict(x) == y for x, y in zip(xs, ys)) / len(ys)

class MajorityClass(Classifier):
    def __init__(self):
        super().__init__("majority")
    def fit(self, xs, ys):
        self.label = max(set(ys), key=ys.count)
    def predict(self, x):
        return self.label

class Threshold(Classifier):
    def __init__(self, cut):
        super().__init__(f"threshold {cut}")
        self.cut = cut
    def predict(self, x):
        return 1 if x >= self.cut else 0

xs, ys = [1, 2, 3, 6, 7, 8], [0, 0, 1, 1, 1, 1]
for model in (MajorityClass(), Threshold(3)):
    model.fit(xs, ys)
    print(model.name, model.accuracy(xs, ys), isinstance(model, Classifier))

majority 0.6666666666666666 True
threshold 3 1.0 True

When Python looks up model.accuracy, it searches the object's class first and then its base classes, so both models find the single shared definition. Threshold does not define fit, so it inherits the empty one.

Use inheritance when the subclass really is a kind of the base class and should work anywhere the base class does: a threshold rule is a classifier. For "has a" relationships use composition, as in the previous section. A good sign that inheritance is wrong is a subclass that has to disable methods it inherited, like the priority queue that should not have insert.

If a base class should never be used on its own, the abc module can say so: decorate the methods every subclass must provide with @abstractmethod, and Python refuses to create an object of a class that has not filled them in.

from abc import ABC, abstractmethod

class Model(ABC):
    @abstractmethod
    def predict(self, x): ...

class Always1(Model):
    def predict(self, x):
        return 1

print(Always1().predict(5))
try:
    Model()
except TypeError:
    print("Model is abstract")

1
Model is abstract

Common mistakes: forgetting to call super().__init__(...), so the base class's attributes are never set; deep chains of subclasses where each adds one small tweak (hard to follow; prefer composition or a parameter); and inheriting purely to reuse code when the "is a" relationship does not hold.

Practice: A Family of Shapes, One Quantile Method for Every Distribution

Dataclasses

A training example for the Machine Learning path is a list of features and a label. A first version writes the class by hand, including the methods that let you print it and compare two examples:

class Example:
    def __init__(self, features, label):
        self.features = features
        self.label = label
    def __repr__(self):
        return f"Example(features={self.features!r}, label={self.label!r})"
    def __eq__(self, other):
        return (self.features, self.label) == (other.features, other.label)

e = Example([1.0, 2.5], "cat")
print(e, e == Example([1.0, 2.5], "cat"))

Example(features=[1.0, 2.5], label='cat') True

Every attribute name appears four times, and adding a field means editing three methods; forget one and == silently ignores the new field. The @dataclass decorator reads the annotated fields in the class body and writes __init__, __repr__ and __eq__ for you:

from dataclasses import dataclass, field

@dataclass
class Example:
    features: list
    label: str
    weight: float = 1.0

e = Example([1.0, 2.5], "cat")
print(e)
print(e == Example([1.0, 2.5], "cat"), e.weight)

@dataclass
class Batch:
    examples: list = field(default_factory=list)   # a fresh list for each Batch

b1, b2 = Batch(), Batch()
b1.examples.append(e)
print(len(b1.examples), len(b2.examples))

Example(features=[1.0, 2.5], label='cat', weight=1.0)
True 1.0
1 0

Each field is a name with a type annotation (features: list); a value after = becomes the default. A mutable default must go through field(default_factory=list), and the dataclass refuses a plain = [], so the shared-list trap from Level 1 cannot happen here.

Two options cover most other needs. frozen=True makes the objects read-only, which also makes them hashable, so they can be dictionary keys and set members, like tuples. order=True adds <, <=, > and >=, comparing the fields in order, just as tuples compare:

from dataclasses import dataclass

@dataclass(frozen=True, order=True)
class Cell:
    row: int
    col: int

visited = {Cell(0, 0), Cell(0, 1)}
print(Cell(0, 1) in visited)
print(sorted([Cell(2, 0), Cell(0, 5), Cell(0, 1)]))
try:
    Cell(0, 0).row = 3
except AttributeError:
    print("frozen: cannot change a Cell")

True
[Cell(row=0, col=1), Cell(row=0, col=5), Cell(row=2, col=0)]
frozen: cannot change a Cell

A frozen dataclass is a tuple with names: as safe as a tuple, and far clearer than cell[0].

Common mistakes: forgetting the type annotation, so the attribute becomes an ordinary class attribute rather than a field; putting a field without a default after one with a default (Python cannot tell which positional argument is which, and raises TypeError); and expecting the annotations to be checked at runtime. Example("oops", 3) works; annotations are documentation for readers and tools (Level 5).

Practice: Race Results as Records

Properties

A Bernoulli trial, from the Statistics path, succeeds with probability p and has variance p(1 - p). A first version works the variance out once, when the object is created:

class Bernoulli:
    def __init__(self, p):
        self.p = p
        self.variance = p * (1 - p)

coin = Bernoulli(0.5)
coin.p = 0.9                # someone updates p...
print(coin.p, coin.variance)
coin.p = 1.7                # ...and nothing stops an impossible value
print(coin.p)

0.9 0.25
1.7

Two bugs in six lines: the variance is stale (still the value for p = 0.5), and p accepts a probability of 1.7. Other languages solve this with get_p() and set_p() methods, which every caller then has to use. Python's @property keeps the plain attribute syntax but runs a method behind it. A read-only property computes its value every time it is read, so it can never go stale, and a setter checks each new value:

class Bernoulli:
    def __init__(self, p):
        self.p = p                      # goes through the setter below

    @property
    def p(self):
        return self._p

    @p.setter
    def p(self, value):
        if not 0 <= value <= 1:
            raise ValueError(f"p must be between 0 and 1, got {value}")
        self._p = value

    @property
    def variance(self):
        return self.p * (1 - self.p)

coin = Bernoulli(0.5)
coin.p = 0.9
print(coin.p, round(coin.variance, 2))
try:
    coin.p = 1.7
except ValueError as e:
    print(e)

0.9 0.09
p must be between 0 and 1, got 1.7

Callers still write coin.p and coin.variance with no brackets, so a class can start with plain attributes and switch to properties later without changing any code that uses it. That is why Python programmers do not write getters and setters in advance.

Use a property for a value that is cheap to compute from the object's state (a vector's length, a rectangle's area, a distribution's variance). If the computation is slow, a plain method makes the cost visible to the caller, and functools.cached_property computes the value once and stores it for objects that never change.

Common mistakes: giving the property and the stored attribute the same name (self.p = value inside the p setter calls the setter again, forever; store it as self._p); calling a property like a method (coin.variance() fails, because coin.variance is already a number); and hiding slow work, such as reading a file, behind something that looks like a cheap attribute.

Practice: A Temperature That Checks Itself, A Window You Can Drag and Resize

Special methods

The Machine Learning path does arithmetic on vectors all the time. A first version of a vector class gives every operation an ordinary method name:

class Vector:
    def __init__(self, *xs):
        self.xs = list(xs)
    def add(self, other):
        return Vector(*[a + b for a, b in zip(self.xs, other.xs)])
    def scale(self, k):
        return Vector(*[k * a for a in self.xs])
    def show(self):
        return "Vector" + str(tuple(self.xs))

u, v = Vector(1, 2), Vector(3, 4)
print(u.add(v).scale(2).show())
print(u == Vector(1, 2))

Vector(8, 12)
False

Formulas turn into chains of method calls, print(u) shows something like <__main__.Vector object at 0x10f3a2b50>, and u == Vector(1, 2) is False because by default == asks whether two names refer to the same object. Python's operators and built-in functions are all hooks: a + b calls a.__add__(b), len(a) calls a.__len__(), print(a) uses a.__repr__(). These special methods (or "dunder" methods, for the double underscores) let your class plug into the language:

class Vector:
    def __init__(self, *xs):
        self.xs = list(xs)
    def __repr__(self):
        return f"Vector{tuple(self.xs)}"
    def __eq__(self, other):
        return isinstance(other, Vector) and self.xs == other.xs
    def __add__(self, other):
        return Vector(*[a + b for a, b in zip(self.xs, other.xs, strict=True)])
    def __mul__(self, k):
        return Vector(*[k * a for a in self.xs])
    __rmul__ = __mul__                       # so that 2 * v works as well as v * 2
    def __len__(self):
        return len(self.xs)
    def __iter__(self):
        return iter(self.xs)

u, v = Vector(1, 2), Vector(3, 4)
print(2 * (u + v))
print(u == Vector(1, 2), len(u))
print(sum(a * b for a, b in zip(u, v)))     # iteration makes the dot product work
x, y = v                                     # ...and so does unpacking
print(x, y, list(u))

Vector(8, 12)
True 2
11
3 4 [1, 2]

The formula 2 * (u + v) now reads like the maths. __iter__ alone makes the object work with for, zip, sum, list, unpacking and in.

__lt__ defines <, and it is what sorted, min, max and heapq use to compare. It lets objects with no natural tuple form go into a heap, such as search states in the Algorithms path ordered by their cost:

import heapq

class State:
    def __init__(self, cost, path):
        self.cost, self.path = cost, path
    def __lt__(self, other):
        return self.cost < other.cost
    def __repr__(self):
        return f"State({self.cost}, {self.path!r})"

frontier = []
heapq.heappush(frontier, State(7, "AC"))
heapq.heappush(frontier, State(2, "AB"))
heapq.heappush(frontier, State(5, "AD"))
print(heapq.heappop(frontier))
print(min([State(4, "x"), State(1, "y")]))

State(2, 'AB')
State(1, 'y')

A good __repr__ is worth writing for every class: it is what you see when printing a list of objects, in error messages and while debugging. The convention is to make it look like the code that would create the object.

Common mistakes: defining __eq__ and then trying to put the objects in a set (defining __eq__ switches off the default hash; add __hash__ too, or use a frozen dataclass); returning a string from __len__ or anything but a whole number; making __add__ change self instead of returning a new object; and defining so many operators that a reader can no longer guess what a @ b % c means. Overload an operator only when its meaning is obvious.

Practice: A Vector Type for the Plane, Measurements That Carry Their Error, A Playlist That Acts Like a List

Protocols and duck typing

A function computes the mean of some values. A first version checks that it has been given a list, to be safe:

def mean(values):
    if not isinstance(values, list):
        raise TypeError("values must be a list")
    return sum(values) / len(values)

print(mean([1, 2, 6]))
try:
    print(mean((1, 2, 6)))
except TypeError as e:
    print(e)

3.0
values must be a list

The check makes the function less useful. A tuple, a range, a deque or the Vector from the previous section would all work perfectly, and the check rejects every one of them. What mean really needs is something it can loop over. Python's approach is called duck typing ("if it walks like a duck and quacks like a duck, it is a duck"): use the object as you need it, and let anything that supports those operations through. The set of methods an operation relies on is called a protocol. for needs the iteration protocol (__iter__); len needs __len__; with (Level 5) needs __enter__ and __exit__.

from collections import deque

def mean(values):
    n = total = 0
    for x in values:           # needs only the iteration protocol
        n += 1
        total += x
    if n == 0:
        raise ValueError("mean of no values")
    return total / n

class Countdown:
    def __init__(self, start):
        self.start = start
    def __iter__(self):
        return iter(range(self.start, 0, -1))

print(mean([1, 2, 6]), mean((1, 2, 6)), mean(range(1, 4)), mean(deque([4, 8])))
print(mean(x * x for x in range(4)), mean(Countdown(5)))
print(list(Countdown(3)), 2 in Countdown(3), sorted(Countdown(3)))

3.0 3.0 2.0 6.0
3.5 3.0
[3, 2, 1] True [1, 2, 3]

This version accepts generators too, which the len-based version could not, because it counts while it loops. The small Countdown class never mentions mean, sorted or in, but it works with all of them because it follows the protocol they rely on.

Duck typing is also why the models in the inheritance section did not strictly need a shared base class: any object with fit and predict methods can go through the same evaluation code. When you want to write down such an expectation, typing.Protocol describes it. It changes nothing when the code runs, but readers and type checkers (Level 5) can see what is required, and a class matches simply by having the methods, with no inheritance:

from typing import Protocol

class Predictor(Protocol):
    def predict(self, x: float) -> int: ...

class AlwaysOne:
    def predict(self, x):
        return 1

def accuracy(model: Predictor, xs, ys):
    return sum(model.predict(x) == y for x, y in zip(xs, ys)) / len(ys)

print(accuracy(AlwaysOne(), [1, 2, 3, 4], [1, 0, 1, 1]))

0.75

Common mistakes: checking isinstance(x, list) when any iterable would do; calling len on an argument that might be a generator; and, in the other direction, relying on duck typing for something that behaves differently in a subtle way. A string is iterable too, so mean("123") fails inside the loop with a confusing TypeError; a clear check with a good message is right when a common mistake needs catching early.

Practice: A Timetable You Can Read Twice, Grading Any Model, Groups, Moves and Zooms

Practice

These problems are required on this path at this level; the path adds one problem from each of the level's topics, and one for each construct the others do not use, picked for you and kept.

Checklist

Write a class with __init__, attributes and methods, and explain what self is.
Build a class out of other objects (composition) and keep its internal state private behind a small interface.
Share behaviour through a base class, override methods, call super().__init__, and choose between "is a" and "has a".
Replace a hand-written __init__, __repr__ and __eq__ with @dataclass, using field(default_factory=...), frozen=True and order=True.
Turn a stale stored value into a computed @property, and validate new values with a setter.
Make objects printable, comparable and iterable with __repr__, __eq__, __lt__, __len__, __iter__ and arithmetic special methods.
Write functions that rely on protocols instead of exact types, and describe an expected interface with typing.Protocol.