Level 1 · Machine Learning Fundamentals: Numbers as Data

Machine learning starts with a translation. Before a program can learn anything about houses, songs, emails or patients, each one has to become a short list of numbers: floor area, number of rooms and age for a house; tempo, loudness and length for a song; how often certain words appear for an email. Once everything is a list of numbers, questions like "which house is most like this one?" or "do these two documents talk about the same thing?" become arithmetic. This level teaches that arithmetic: vectors, the dot product, length, distance and similarity, and how to find the closest example in a dataset.

You will write every operation yourself with plain Python lists. A vector is a list of numbers, and a dataset is a list of those lists. Libraries exist that do all of this faster, but they hide exactly the ideas this path is about, and every one of the functions below is only a few lines long. Keep the examples small enough to check by eye: if you cannot predict what a function prints for a three-number vector, it is not ready for a thousand.

The ideas here connect to the other paths. The average of a column of numbers is the mean from Statistics level 1, and here it becomes the "centre" of a set of vectors. Finding the closest vector is a linear scan with min(..., key=...), the same one-pass pattern you use throughout the Algorithms path, and in Level 2 it becomes your first learning algorithm, the nearest-neighbour rule.

Vectors as lists

A vector is an ordered list of numbers. Each position means something fixed: in the house data below, position 0 is always the floor area in square metres, position 1 the number of rooms and position 2 the age in years. Each position is called a feature, and the number of positions is the vector's dimension. A dataset is a list of vectors that all have the same dimension, one per example; you can picture it as a table where each row is an example and each column is a feature.

houses = [
    [70, 3, 25],    # area in m², rooms, age in years
    [120, 5, 5],
    [55, 2, 40],
    [95, 4, 12],
]
print("examples:", len(houses))
print("dimension:", len(houses[0]))
print("second house:", houses[1])
print("its age:", houses[1][2])
areas = [h[0] for h in houses]
print("area column:", areas)

examples: 4
dimension: 3
second house: [120, 5, 5]
its age: 5
area column: [70, 120, 55, 95]

Reading a row is plain indexing, houses[1]. Reading a column, all the values of one feature, needs a loop or a comprehension over the rows, [h[0] for h in houses]. Most of the work in this path is one of those two shapes: something done to each row, or something computed down each column.

Because nothing in Python forces every row to have the same length, it is worth checking once, when the data arrives, rather than discovering a short row halfway through a calculation:

def check_dataset(rows):
    d = len(rows[0])
    for i, row in enumerate(rows):
        if len(row) != d:
            raise ValueError(f"row {i} has {len(row)} features, expected {d}")
    return d

print(check_dataset([[1, 2], [3, 4], [5, 6]]))
try:
    check_dataset([[1, 2], [3, 4, 0], [5, 6]])
except ValueError as e:
    print("error:", e)

2
error: row 1 has 3 features, expected 2

Common mistakes: mixing up rows and columns (houses[0] is the first house, not the first feature). Storing features in a different order in different rows, which no program can detect for you: decide the order once and write it in a comment. Using a list of strings like ["70", "3"] read from a file without converting to numbers, so + joins text instead of adding.

Practice: The Range of Every Feature

Element-wise operations

Adding two vectors means adding them position by position: [1, 2, 3] + [10, 20, 30] should be [11, 22, 33]. Python's + on lists does something else entirely (it joins them), so you write the operation yourself. zip pairs up the positions, and a comprehension builds the result:

def add(u, v):
    return [a + b for a, b in zip(u, v)]

def sub(u, v):
    return [a - b for a, b in zip(u, v)]

def scale(c, v):
    return [c * a for a in v]

u, v = [1, 2, 3], [10, 20, 30]
print(u + v)          # list concatenation, not vector addition
print(add(u, v))
print(sub(v, u))
print(scale(0.5, v))

[1, 2, 3, 10, 20, 30]
[11, 22, 33]
[9, 18, 27]
[5.0, 10.0, 15.0]

These three operations are enough to express a surprising amount. The mean vector (or centroid) of a dataset is the sum of all rows scaled by one over the number of rows. Each position of the result is the mean of one column, exactly the mean from Statistics level 1, computed for every feature at once. For n rows \mathbf{x}_1, \dots, \mathbf{x}_n it is

def mean_vector(rows):
    total = [0] * len(rows[0])
    for row in rows:
        total = add(total, row)
    return scale(1 / len(rows), total)

houses = [[70, 3, 25], [120, 5, 5], [55, 2, 40], [95, 4, 12]]
print(mean_vector(houses))

[85.0, 3.5, 20.5]

The centroid is the "typical" house of this dataset: 85 m², 3.5 rooms, 20.5 years old. No real house has 3.5 rooms, and that is fine; a centroid is a summary, not an example. Subtracting the centroid from every row, sub(row, centre), is called centring the data: afterwards each feature has mean zero, and a positive number simply means "more than typical". Centring comes back when you scale features in Level 3 and when you find the main directions of a dataset with PCA in Level 6.

Common mistakes: zip silently stops at the shorter list, so add([1, 2, 3], [1, 2]) returns a two-element vector instead of failing. If your vectors might differ in length, check with len first, or use zip(u, v, strict=True), which raises an error. Another common slip is writing u + v out of habit and getting a list twice as long.

Practice: Compared with the Typical Member, Colour Steps Between Two Paints

The dot product

The dot product of two vectors of the same dimension multiplies them position by position and adds up the results: dot([1, 2, 3], [4, 5, 6]) = 1·4 + 2·5 + 3·6 = 32. In symbols, for two vectors \mathbf{u} and \mathbf{v} of dimension n, with entries u_i and v_i,

It is the single most used operation in machine learning, and it is one line of Python. The sum becomes sum(...), the index i becomes the for, and zip hands over u_i and v_i together:

def dot(u, v):
    return sum(a * b for a, b in zip(u, v))

print(dot([1, 2, 3], [4, 5, 6]))
print(dot([1, 0], [0, 1]))       # perpendicular directions
print(dot([2, 3], [-2, -3]))     # opposite directions

32
0
-13

There are two ways to think about it, and both matter.

The first is a weighted sum. If one vector holds features and the other holds a weight for each feature, the dot product is a score that adds up each feature times how much it counts. Suppose an estate agent's rule of thumb prices a house at 3,000 per square metre, plus 10,000 per room, minus 1,000 per year of age. The weights are [3000, 10000, -1000] and the price of every house is one dot product:

weights = [3000, 10000, -1000]
houses = [[70, 3, 25], [120, 5, 5], [55, 2, 40], [95, 4, 12]]
for h in houses:
    print(h, dot(weights, h))

[70, 3, 25] 215000
[120, 5, 5] 405000
[55, 2, 40] 145000
[95, 4, 12] 313000

That is a linear model: a prediction that is a weighted sum of the features, \hat{y} = \mathbf{w}\cdot\mathbf{x}, where \mathbf{w} holds the weights, \mathbf{x} the features and \hat{y} (read "y hat") is the prediction. From Level 3 on it usually gets a fixed extra term, the bias b: \hat{y} = \mathbf{w}\cdot\mathbf{x} + b. In this example someone chose the weights by hand. From Level 3 onwards you will learn them from data, and in Level 4 you will see that learning them means searching for the weights that make the predictions least wrong.

The second way is geometric. The dot product measures how much two vectors point the same way. It is positive when they point roughly the same way, zero when they are perpendicular, and negative when they point roughly opposite ways, as the first example showed. The next sections make this precise with lengths and angles.

Common mistakes: taking the dot product of vectors of different lengths (again, zip hides it). Confusing the dot product, which gives one number, with element-wise multiplication, which gives a vector: [a * b for a, b in zip(u, v)] is a list, sum of it is the dot product.

Practice: Weighted Scores at the Bake-Off, Tailwind, Headwind or Crosswind

Length (the norm)

The length of a vector, also called its norm, comes from Pythagoras. In two dimensions the vector [3, 4] reaches from the origin to the point (3, 4), and its length is √(3² + 4²) = 5. In any dimension the length is the square root of the sum of the squares, which is the square root of the vector's dot product with itself:

import math

def norm(v):
    return math.sqrt(dot(v, v))

print(norm([3, 4]))
print(norm([1, 2, 2]))
print(norm([0, 0, 0]))
print(math.hypot(1, 2, 2))    # the standard library agrees

5.0
3.0
0.0
3.0

This is the Euclidean norm, written \lVert \mathbf{v} \rVert or called the L2 norm. There is a second norm you will meet often: the Manhattan norm (L1 norm), the sum of the absolute values, \lVert \mathbf{v} \rVert_1 = \sum_{i=1}^{n} |v_i|. It is how far you walk on a street grid where you can only move along the axes.

def norm1(v):
    return sum(abs(a) for a in v)

print(norm1([3, 4]), norm([3, 4]))
print(norm1([3, -4]), norm([3, -4]))

7 5.0
7 5.0

The L1 length of [3, 4] is 7 and the L2 length is 5: walking round the block is longer than cutting across. Both norms are zero only for the zero vector, and both double when you double the vector. Scaling a vector by c multiplies its length by abs(c), a fact the section on normalising relies on.

Common mistakes: forgetting the square root and using dot(v, v) as the length (that is the squared length, which is often what you want when comparing, but it is not the length). Writing sum(v) for the L1 norm, which lets negative entries cancel positive ones: sum([3, -4]) is -1, not 7.

Practice: A Speed Limit That Keeps the Heading

Euclidean and Manhattan distance

The distance between two vectors is the length of their difference. With the Euclidean norm that is the straight-line distance; with the Manhattan norm it is the grid-walking distance:

Both are built from functions you already have:

def euclidean(u, v):
    return norm(sub(u, v))

def manhattan(u, v):
    return norm1(sub(u, v))

p, q = [1, 2], [4, 6]
print(euclidean(p, q), manhattan(p, q))
print(math.dist(p, q))        # standard library check for the Euclidean one
print(euclidean(p, p))        # every vector is at distance 0 from itself

5.0 7
5.0
0.0

A distance turns "how similar are these two examples?" into a number, and that is what nearest-neighbour methods, clustering and anomaly detection in later levels are built on. Any sensible distance is zero from a point to itself, is the same in both directions, and obeys the triangle inequality: going from a to c directly is never longer than going via b.

The two distances usually agree about what is near and what is far, but not always. Which of a and b is closer to the origin?

origin, a, b = [0, 0], [3, 3], [0, 5]
print("euclidean:", round(euclidean(origin, a), 2), euclidean(origin, b))
print("manhattan:", manhattan(origin, a), manhattan(origin, b))

euclidean: 4.24 5.0
manhattan: 6 5

In straight lines a is closer; walking the grid, b is. Euclidean distance punishes one large difference more than several small ones (because the differences are squared before adding), while Manhattan distance treats every unit of difference the same. Neither is "correct": choosing the distance is part of designing the model, and Level 3 shows that the units of the features matter even more than the formula.

Common mistakes: comparing distances between vectors whose features are on wildly different scales (an area in square metres next to a number of rooms), where the large feature decides everything; this is the feature scaling problem of Level 3. Taking square roots you do not need: when you only want to know which of two points is closer, comparing squared Euclidean distances gives the same answer and is cheaper.

Practice: Distance Between Two Points, As the Crow Flies, or by Taxi, The Office Everyone Can Reach

From formula to Python

Machine learning books write these operations as formulas, and nearly every formula in this level is a sum. A sum turns into Python in a fixed way: the summation index becomes the for, the range of the index becomes what you loop over, and the expression after the \sum becomes the expression in front of for. When the formula walks along two vectors at once, u_i next to v_i, zip does the walking.

The last row is a third length, the largest single entry, sometimes called the max norm; max takes the place of \sum and the pattern is otherwise the same. Writing each formula as one comprehension, without the helper functions, gives the same numbers:

import math
u, v = [1, 2], [4, 6]
print(sum(a * b for a, b in zip(u, v)))
print(math.sqrt(sum((a - b) ** 2 for a, b in zip(u, v))))
print(sum(abs(a - b) for a, b in zip(u, v)))
print(max(abs(a) for a in v))

16
5.0
7
6

Read in the other direction, this is also how to check a formula in a paper: if you cannot say what the for would loop over, you have not yet understood which symbol is the index.

Practice: Three Ways to Measure a Move

Cosine similarity

Distance compares where two vectors end. Sometimes what matters is only the direction they point in. Documents are the classic case. Represent each document by how many times it uses each word of a small vocabulary; a long article about football and a short match report use the same words in roughly the same proportions, but the long one has much bigger counts.

vocab = ["ball", "goal", "team", "vote", "party"]
report  = [2, 3, 1, 0, 0]
article = [8, 10, 5, 0, 1]
speech  = [0, 0, 1, 5, 4]
print(round(euclidean(report, article), 2), round(euclidean(report, speech), 2))

10.1 7.35

By Euclidean distance the short match report is closer to the political speech than to the long football article, simply because both short documents have small counts. The fix is to compare angles. The dot product and the lengths are linked by

where \theta is the angle between the vectors. Rearranged, that gives the cosine similarity:

def cosine(u, v):
    return dot(u, v) / (norm(u) * norm(v))

print(round(cosine(report, article), 3))
print(round(cosine(report, speech), 3))
print(round(cosine(report, scale(10, report)), 3))   # same direction, ten times longer

0.989
0.041
1.0

Cosine similarity is 1 for vectors pointing the same way, 0 for perpendicular ones and -1 for opposite ones. Scaling either vector does not change it, which is exactly the property the documents needed: the report and the article now look alike (0.99), the report and the speech do not (0.04). Word counts are never negative, so for documents the cosine stays between 0 and 1. In Level 6 you will use cosine similarity to search a collection of embeddings, vectors that represent words or items so that similar meanings point in similar directions.

A related number is the cosine distance, 1 - cosine(u, v), used when an algorithm expects "smaller means closer". It is not a true distance (it can break the triangle inequality), but it ranks neighbours the same way cosine similarity does, only in reverse.

Common mistakes: dividing by zero when one of the vectors is all zeros (an empty document); decide what that should mean, usually similarity 0, and handle it before dividing. Expecting cosine similarity to notice size differences: a price vector and the same prices doubled have similarity 1.

Practice: Do These Two Reviews Agree?, Columns That Say the Same Thing

Normalising a vector

To normalise a vector is to scale it to length 1 while keeping its direction: divide every entry by the vector's length, \hat{\mathbf{v}} = \mathbf{v} / \lVert \mathbf{v} \rVert. The result is a unit vector.

def normalise(v):
    n = norm(v)
    if n == 0:
        raise ValueError("cannot normalise the zero vector")
    return scale(1 / n, v)

u = normalise([3, 4])
print(u, norm(u))
print([round(a, 3) for a in normalise([2, 3, 1, 0, 0])])

[0.6000000000000001, 0.8] 1.0
[0.535, 0.802, 0.267, 0.0, 0.0]

The 0.6000000000000001 is ordinary floating-point rounding: 1 / 5 cannot be stored exactly in binary, so 0.2 * 3 is a hair above 0.6. Such tiny errors are harmless, but they are why the examples in this path round before printing and never test floats with ==.

Unit vectors make cosine similarity cheap. For unit vectors the lengths in the cosine formula are both 1, so the cosine is the dot product. If you are going to compare one query against thousands of stored vectors, normalise everything once, and each comparison becomes a single dot:

docs = {"report": [2, 3, 1, 0, 0], "article": [8, 10, 5, 0, 1], "speech": [0, 0, 1, 5, 4]}
unit = {name: normalise(v) for name, v in docs.items()}
query = normalise([1, 1, 0, 0, 0])      # "ball goal"
for name, v in unit.items():
    print(name, round(dot(query, v), 3))

report 0.945
article 0.923
speech 0.0

Normalised vectors also show why Euclidean distance and cosine similarity are close relatives. For two unit vectors, the squared distance between them is \lVert \mathbf{u} - \mathbf{v} \rVert^2 = 2 - 2\cos\theta, so the nearest unit vector by distance is also the most similar one by angle. Normalising first and then using ordinary distance is a standard way to get cosine behaviour out of a method that only knows about distances.

A word of warning about the name. "Normalising" is used for two different things in machine learning. Here it means scaling one vector (a row) to length 1. In Level 3 you will also normalise features (columns), so that every column has, say, mean 0 and standard deviation 1, which is the z-score from Statistics level 2. Both fix a scale problem, but they are different operations; say which one you mean.

Common mistakes: normalising the zero vector (there is no direction to keep, and you divide by zero). Normalising in place while still needing the original lengths. Comparing a normalised query with unnormalised stored vectors and calling the dot product a cosine.

Practice: The Average Wind Direction

The closest vector in a dataset

Everything so far compares two vectors. The last step of this level compares one vector, the query, against a whole dataset and returns the closest one. The plain version is a loop that remembers the best so far:

def closest(query, rows):
    best_i, best_d = None, math.inf
    for i, row in enumerate(rows):
        d = euclidean(query, row)
        if d < best_d:
            best_i, best_d = i, d
    return best_i, best_d

points = [[1, 1], [4, 5], [6, 2], [2, 4]]
print(closest([5, 3], points))

(2, 1.4142135623730951)

To see the answer, draw the points and the query. Press Run to see them; the query [5, 3] sits next to [6, 2]:

points = [[1, 1], [4, 5], [6, 2], [2, 4]]
plot([p[0] for p in points], [p[1] for p in points], kind="scatter", label="dataset")
plot([5], [3], kind="scatter", label="query")

This is the one-pass minimum from the Algorithms path, and Python's min with a key expresses it directly: "the index whose row has the smallest distance to the query".

def closest_index(query, rows, dist=euclidean):
    return min(range(len(rows)), key=lambda i: dist(query, rows[i]))

print(closest_index([5, 3], points))
print(closest_index([5, 3], points, dist=manhattan))
print(closest_index([3, 2], points))    # a tie: [1, 1] and [2, 4] are both at √5

2
2
0

When several rows are equally close, min returns the first one it meets. That is a fine rule, but it is a rule, and it can change your predictions if you reorder the data; make ties deliberate. The dist parameter lets you swap in any distance, including lambda u, v: -cosine(u, v) for "most similar by angle".

Often you want the k closest rows rather than just one. Sorting all the indices by distance and taking the first k works; heapq.nsmallest does the same job without sorting everything, which matters when the dataset is large and k is small (heaps are an Algorithms path topic, and this is one of their most common uses):

import heapq

query = [5, 3]
by_distance = sorted(range(len(points)), key=lambda i: euclidean(query, points[i]))
print(by_distance[:2])
print(heapq.nsmallest(2, range(len(points)), key=lambda i: euclidean(query, points[i])))

[2, 1]
[2, 1]

Scanning the whole dataset costs one distance per row, so a query against n rows of dimension d takes about n·d steps. For a few thousand rows that is instant. For millions of rows and many queries, real systems use search structures (k-d trees, or approximate methods that give up exactness for speed), a topic that returns in Level 6 with similarity search.

Why does this matter for learning? Suppose every row of the dataset comes with an answer: a price for each house, a "spam" or "not spam" for each email. Then the answer attached to the closest row is a reasonable guess for the query. That is the nearest-neighbour rule, the first learning algorithm of Level 2.

Common mistakes: returning the closest row when you later need its index (to look up its label), or the other way round; returning the index is usually more useful. Forgetting that the query itself may be in the dataset, so its closest neighbour is itself at distance 0. Starting the best distance at 0 instead of math.inf, so no row ever beats it.

Practice: Songs Within Reach, Every Nearest Charger, Ties Included, Sorting Fruit by the Typical Example

Practice

These problems are required on this path at this level; the path adds one problem from each of the level's topics, picked for you and kept.

• As the Crow Flies, or by Taxi
• Songs Within Reach
• Do These Two Reviews Agree?
• The Range of Every Feature
• Colour Steps Between Two Paints
• A Speed Limit That Keeps the Heading

Checklist

• Store examples as vectors (lists) and a dataset as a list of rows, and read a row or a column.
• Add, subtract and scale vectors element-wise, and compute the mean vector (centroid) of a dataset.
• Compute a dot product and explain it both as a weighted sum (a linear model's score) and as a measure of direction.
• Compute Euclidean (L2) and Manhattan (L1) lengths and distances, and give an example where they disagree about which point is closer.
• Compute cosine similarity, explain why it ignores length, and say when that is what you want.
• Normalise a vector to length 1, handle the zero vector, and use unit vectors to turn cosine similarity into a dot product.
• Find the closest vector, and the k closest vectors, to a query with min, sorted or heapq.nsmallest and a key.
• Explain how the closest labelled example could be used to make a prediction.