Level 1 · Statistics & Probability: Describing Data

Statistics starts with a pile of numbers and one question: what do they say? Before anyone can test a claim, fit a model or quote a probability, somebody has to look at the data and summarise it honestly. This level is about those first summaries: how many values there are, where the middle is, what is typical, how far the values stretch, and how often each one occurs. You will compute every one of them yourself, with a loop or a dictionary, and only then compare with Python's statistics module.

A summary is a compression. Fifteen numbers become one, and something is always lost. So each section ends with the same question: when does this number mislead? A mean can be dragged far from every real value by one outlier, a percentage can hide a tiny sample, and a range can be decided by a single typo. Knowing which summary to trust, and saying what it means in plain words, matters as much as computing it.

The paths overlap here. In the Algorithms path the mean and the maximum are one-pass computations (visit each element once, keep a running total) and the median is a reason to sort. In the Machine Learning path the same mean becomes the centre used to normalise a feature. Here the question is different: not how fast, but what the number tells you about the world.

Counting and summing

A dataset, for now, is a Python list. Here are the number of minutes a bus arrived late on fifteen school days (0 means on time):

late = [3, 0, 5, 2, 0, 4, 1, 0, 3, 27, 2, 0, 6, 3, 1]

n = 0
total = 0
for x in late:
    n += 1
    total += x
print("days:", n)
print("total minutes late:", total)
print("check:", len(late), sum(late))

days: 15
total minutes late: 57
check: 15 57

Counting and summing are the building blocks of almost everything that follows. The loop keeps two running values and updates them once per element: this is the one-pass pattern from the Algorithms path, and it works even when the data is too long to hold in memory, because each value is looked at once and then forgotten.

Often you do not want every value but the ones that satisfy a condition. Counting "how many days was the bus at least 5 minutes late?" is a sum of ones:

late = [3, 0, 5, 2, 0, 4, 1, 0, 3, 27, 2, 0, 6, 3, 1]

count = 0
for x in late:
    if x >= 5:
        count += 1
print(count)
print(sum(1 for x in late if x >= 5))
print(sum(x >= 5 for x in late))

3
3
3

The last line works because True counts as 1 and False as 0 when added. It is short, but the middle version says what it means more clearly.

Common mistakes: starting total inside the loop, so it resets every time. Using len(data) on a generator (it has no length; count as you go). Dividing before the loop has finished. And mixing units: if some values are minutes and others seconds, the sum is meaningless even though Python computes it happily.

Practice: The Ferry Delay Log, Dice Tally

The mean

The mean (the everyday "average") is the total divided by the count. It answers: if the total were shared out equally, how much would each value get? Written as a formula, for n values x_1, x_2, \dots, x_n:

\bar{x} (read "x bar") is the usual name for the mean, and \sum_{i=1}^{n} x_i means "add up x_i for every i from 1 to n", which is the running total of the loop below.

def mean(data):
    total = 0
    n = 0
    for x in data:
        total += x
        n += 1
    return total / n

late = [3, 0, 5, 2, 0, 4, 1, 0, 3, 27, 2, 0, 6, 3, 1]
print(mean(late))

3.8

So on average the bus was 3.8 minutes late. Notice that no single day was 3.8 minutes late: the mean is a property of the whole collection, not a typical member of it.

The mean has a neat physical picture. If you put equal weights on a ruler at each value, the mean is the point where the ruler balances. In numbers: the deviations x - mean always add up to zero. Values above the mean pull one way, values below pull the other, and they cancel exactly.

late = [3, 0, 5, 2, 0, 4, 1, 0, 3, 27, 2, 0, 6, 3, 1]
m = sum(late) / len(late)
deviations = [x - m for x in late]
print(round(sum(deviations), 10))
print(sorted(round(d, 1) for d in deviations))

0.0
[-3.8, -3.8, -3.8, -3.8, -2.8, -2.8, -1.8, -1.8, -0.8, -0.8, -0.8, 0.2, 1.2, 2.2, 23.2]

(round(…, 10) hides a tiny floating-point leftover such as 1e-15.) Look at the deviations: fourteen days sit between about 4 below and 2 above the mean, and one day, the 27, sits 23 above it. That single day holds the ruler up on its own. Remove it and see:

late = [3, 0, 5, 2, 0, 4, 1, 0, 3, 27, 2, 0, 6, 3, 1]
without = [x for x in late if x != 27]
print(round(sum(late) / len(late), 2))
print(round(sum(without) / len(without), 2))

3.8
2.14

One value out of fifteen moved the mean by more than one and a half minutes. The mean is sensitive to outliers: a single extreme value, whether a real breakdown or a typing error (27 instead of 2?), can move it a long way. Before you report a mean, look at the data for values like that and decide what they are.

Weighted means. Sometimes the values you have are already averages of groups of different sizes. Suppose class A (10 students) averaged 80 on a test and class B (30 students) averaged 60. The mean over all 40 students is not (80 + 60) / 2 = 70, because B has three times as many students:

groups = [(80, 10), (60, 30)]   # (group mean, group size)
total = sum(m * size for m, size in groups)
count = sum(size for m, size in groups)
print(total / count)
print((80 + 60) / 2)

65.0
70.0

Multiplying each group mean by its size recovers the group's total, and then it is an ordinary mean again. The same idea gives a quick way to update a mean when one new value arrives: new_mean = old_mean + (x - old_mean) / new_n. That formula never stores the data, so it works on a stream of values of any length, in the spirit of the one-pass loops of the Algorithms path; in Level 4 it becomes the law of large numbers in action.

Common mistakes: averaging averages without weighting by group size. Reporting a mean for data with a wild outlier without saying so. Taking the mean of an empty list (a ZeroDivisionError: decide what an empty group should report). And taking the mean of codes that are not quantities: the mean of postcodes or of shirt numbers is a number, but it means nothing.

Practice: Average Score, Merging the Branch Averages, What Do I Need on the Last Tests?

From formula to Python

Statistics writes its recipes as formulas, and most of them are sums. A sum turns into Python in the same way every time: the index under the \sum becomes the for, the range of the index becomes what you loop over, and the expression after the \sum (the summand) becomes the expression in front of for. Wrap it in sum(...) and you have a generator expression. Python loops over the values themselves, so for x in xs stands for "for i from 1 to n", and x plays the part of x_i.

Here is everything from this level so far, side by side:

The bracket [x_i \ge 5] is 1 when the condition holds and 0 when it does not, which is exactly how Python adds up True and False. In the weighted mean, m_g is the mean of group g and n_g its size. The last row is a recurrence: each new mean \bar{x}_k is built from the previous one, so in Python it is a loop that updates one variable.

late = [3, 0, 5, 2, 0, 4, 1, 0, 3, 27, 2, 0, 6, 3, 1]
m = sum(late) / len(late)
print(m)
print(sum(x >= 5 for x in late))

groups = [(80, 10), (60, 30)]   # (group mean, group size)
print(sum(ng * mg for mg, ng in groups) / sum(ng for mg, ng in groups))
print(round(sum(x - m for x in late), 10))

running = 0
for k, x in enumerate(late, start=1):
    running += (x - running) / k
print(round(running, 10))

3.8
3
65.0
0.0
3.8

The same pattern runs through the whole path. A product \prod becomes math.prod(...), a largest value \max_i becomes max(...), and a sum over pairs \sum_i x_i y_i becomes sum(x * y for x, y in zip(xs, ys)). Level 2 writes the variance this way, Level 4 the expected value, and Level 6 the log-likelihood. When a formula looks hard, read it from the inside out: first the summand, then what it runs over, then what is done with the total.

Practice: Compared with the Typical Member, Above the Class Average

The median

The median is the middle value once the data is in order: half the values are at or below it, half at or above. To find it, sort first. With an odd count there is one middle value; with an even count there are two, and the convention is to take their mean.

def median(data):
    s = sorted(data)
    n = len(s)
    mid = n // 2
    if n % 2 == 1:
        return s[mid]
    return (s[mid - 1] + s[mid]) / 2

late = [3, 0, 5, 2, 0, 4, 1, 0, 3, 27, 2, 0, 6, 3, 1]
print(sorted(late))
print(median(late))
print(median([4, 1, 3, 2]))

[0, 0, 0, 0, 1, 1, 2, 2, 3, 3, 3, 4, 5, 6, 27]
2
2.5

With fifteen values the middle one is the 8th, at index 7 (indices start at 0). With four values the middle pair is 2 and 3, so the median is 2.5, a value that does not occur in the data. That is normal.

The median ignores how far away the extreme values are; it only cares which side of the middle they fall on. Change the 27 to 270 or to 7 and the median stays at 2, while the mean jumps around:

def median(data):
    s = sorted(data)
    n = len(s)
    mid = n // 2
    return s[mid] if n % 2 == 1 else (s[mid - 1] + s[mid]) / 2

for worst in (7, 27, 270):
    data = [3, 0, 5, 2, 0, 4, 1, 0, 3, worst, 2, 0, 6, 3, 1]
    print(worst, "mean", round(sum(data) / len(data), 2), "median", median(data))

7 mean 2.47 median 2
27 mean 3.8 median 2
270 mean 20.0 median 2

This property is called robustness: the median resists outliers. It is why house prices and incomes are usually reported as medians. A handful of mansions or billionaires pulls the mean up so far that it describes nobody, while the median still describes the household in the middle.

Mean and median together tell you about the shape of the data. When the mean is well above the median, as here (3.8 against 2), a few large values stretch the data out to the right: the data is right-skewed. Waiting times, incomes and file sizes are typically like that. When the two are close, the data is roughly symmetric.

A link to the Algorithms path: sorting costs O(n log n), which is fine for thousands of values. For millions there is quickselect, which finds the k-th smallest value in O(n) on average without sorting everything. For a lesson on describing data, sorted is exactly right. Python's own version agrees with ours:

import statistics
late = [3, 0, 5, 2, 0, 4, 1, 0, 3, 27, 2, 0, 6, 3, 1]
print(statistics.median(late), statistics.mean(late))
print(statistics.median([4, 1, 3, 2]))

2 3.8
2.5

Common mistakes: forgetting to sort (the middle of the unsorted list is just whichever day happened to be recorded eighth). Using n / 2 as an index, which is a float and raises a TypeError; use n // 2. Off-by-one errors in the even case: the middle pair is at mid - 1 and mid. And sorting the caller's list in place with data.sort(), which quietly reorders their data; sorted(data) makes a copy.

Practice: The Middle House Price, Which Way Does the Data Lean?, One Typo, Two Averages

The mode

The mode is the most common value. It is the only one of the three "averages" that works for data that is not numbers at all: you cannot take the mean of "bus", "bike" and "walk", but you can say which occurs most often. To find it, count each value in a dictionary, then pick the largest count.

late = [3, 0, 5, 2, 0, 4, 1, 0, 3, 27, 2, 0, 6, 3, 1]

counts = {}
for x in late:
    counts[x] = counts.get(x, 0) + 1
print(counts)

best = max(counts.values())
modes = [x for x, c in counts.items() if c == best]
print("highest count:", best, "modes:", modes)

{3: 3, 0: 4, 5: 1, 2: 2, 4: 1, 1: 2, 27: 1, 6: 1}
highest count: 4 modes: [0]

The bus was on time (0 minutes late) on four days, more often than any other value. Notice that the code returns a list of modes. Data can have several values tied for the top count, and choosing one of them silently (which is what max(counts, key=counts.get) does) hides the tie:

travel = ["bus", "walk", "bike", "bus", "car", "walk", "bus", "walk", "bike"]
counts = {}
for t in travel:
    counts[t] = counts.get(t, 0) + 1
best = max(counts.values())
print(counts)
print([t for t, c in counts.items() if c == best])

{'bus': 3, 'walk': 3, 'bike': 2, 'car': 1}
['bus', 'walk']

Here "bus" and "walk" are tied with three each: the data is bimodal. Python's statistics.multimode returns all of them, while statistics.mode returns only the first one it met. In the Python Fluency path you will meet collections.Counter, which does this counting in one line; writing the dictionary loop yourself first shows that there is nothing magic inside it.

The mode is most useful for categories and for small whole numbers (number of siblings, shoe sizes). For measured quantities such as heights in centimetres to two decimals, almost every value is different, every count is 1, and "the mode" is meaningless. The fix is to group the values into ranges first, which is what a frequency table does (two sections below).

Common mistakes: reporting one mode when there is a tie. Taking the mode of continuous measurements without grouping them. Confusing the mode (the value) with its count (how often it occurs): the mode of the lateness data is 0, not 4.

Practice: Every Favourite, Ties Included, Most Common Value, Smallest Wins Ties

Minimum, maximum and range

The range is the largest value minus the smallest. It is the simplest measure of spread: how far apart the values are. Both ends can be found in one pass, keeping the smallest and largest seen so far:

late = [3, 0, 5, 2, 0, 4, 1, 0, 3, 27, 2, 0, 6, 3, 1]

lo = hi = late[0]
for x in late[1:]:
    if x < lo:
        lo = x
    if x > hi:
        hi = x
print("min", lo, "max", hi, "range", hi - lo)
print(min(late), max(late))

min 0 max 27 range 27
0 27

Starting with the first value (rather than, say, 0) matters: if every value were negative, a starting maximum of 0 would be wrong. This is the same running-best pattern you used in the Algorithms path.

The range says "the lateness varied over 27 minutes", which is true but not very informative, because it depends on just two values, and here one of them is the outlier. Two datasets can have the same mean and the same range and look completely different:

steady = [5, 5, 5, 5, 5, 5, 0, 10]
spread = [0, 0, 0, 0, 10, 10, 10, 10]
for name, d in (("steady", steady), ("spread", spread)):
    print(name, "mean", sum(d) / len(d), "range", max(d) - min(d))

steady mean 5.0 range 10
spread mean 5.0 range 10

In steady most values sit at the mean and two stray to the edges; in spread every value is at an edge. The range cannot tell them apart. Level 2 introduces the standard deviation and the interquartile range, which measure how far values typically are from the centre and are much less at the mercy of a single value.

Minimum and maximum are still worth printing every time you meet a new dataset. They are the fastest way to spot impossible values: a negative age, a 25-hour day, a temperature of 999 used as a "missing" marker.

Common mistakes: initialising the running maximum to 0 or the running minimum to a "big" number that turns out not to be big enough; start with the first element (or math.inf). Calling min or max on an empty list (a ValueError). And reading too much into the range: it grows as you collect more data, simply because more data gives rare extremes more chances to appear.

Practice: Temperature Swing, The Range of Every Feature, Clean the Log, Then Summarise

Frequency tables

A frequency table lists each value (or group of values) with how often it occurs. For measurements with many distinct values, you first put them into bins: equal-width intervals such as 50–59, 60–69 and so on. Integer division does the binning: score // 10 * 10 maps 67 to 60 and 70 to 70.

scores = [67, 82, 45, 91, 73, 58, 77, 88, 62, 70,
          79, 84, 55, 68, 93, 71, 76, 64, 81, 74]

table = {}
for s in scores:
    b = s // 10 * 10
    table[b] = table.get(b, 0) + 1

for b in sorted(table):
    print(f"{b}-{b + 9}: {table[b]:2d} {'#' * table[b]}")

40-49:  1 #
50-59:  2 ##
60-69:  4 ####
70-79:  7 #######
80-89:  4 ####
90-99:  2 ##

The row of # marks is a sideways histogram. Even in text it shows at once what the list of twenty numbers hid: most scores are in the 60s, 70s and 80s, with the 70s the most common bin, and a single low score in the 40s.

The site can also draw it as a chart. Give plot the bin labels and the counts with kind="bar", and press Run to see the histogram:

scores = [67, 82, 45, 91, 73, 58, 77, 88, 62, 70,
          79, 84, 55, 68, 93, 71, 76, 64, 81, 74]
starts = range(40, 100, 10)
counts = [sum(b <= s < b + 10 for s in scores) for b in starts]
plot([f"{b}-{b + 9}" for b in starts], counts, kind="bar", xlabel="score", ylabel="students")

Each count is the bracket sum from the formula section: for each bin, add up 1 for every score that falls inside it.

Counts are hard to compare between groups of different sizes, so tables usually add the relative frequency (the count as a fraction or percentage of the total) and the cumulative percentage (the share at or below the end of each bin):

scores = [67, 82, 45, 91, 73, 58, 77, 88, 62, 70,
          79, 84, 55, 68, 93, 71, 76, 64, 81, 74]

table = {}
for s in scores:
    b = s // 10 * 10
    table[b] = table.get(b, 0) + 1

n = len(scores)
running = 0
print("bin     count  pct  cum")
for b in range(40, 100, 10):
    c = table.get(b, 0)
    running += c
    print(f"{b}-{b + 9}  {c:5d} {100 * c / n:4.0f}% {100 * running / n:4.0f}%")

bin     count  pct  cum
40-49      1    5%    5%
50-59      2   10%   15%
60-69      4   20%   35%
70-79      7   35%   70%
80-89      4   20%   90%
90-99      2   10%  100%

Two details make this table honest. The loop runs over every bin from 40 to 90, so an empty bin would print a count of 0 rather than vanish; a missing row is easy to overlook, and it can be the most interesting fact in the table. And the cumulative column ends at 100%, a quick check that nothing was lost.

The cumulative column answers questions like "what share of students scored below 70?" directly: 35% (the cumulative value at the end of the 60s bin). The bin width is a choice. Too wide (one bin 0–99) hides everything; too narrow (one bin per score) gives a row of ones, the same problem as the mode of continuous data. A handful of bins, roughly the square root of the number of values, is a sensible start.

Common mistakes: unequal bin widths, which make a wide bin look popular just because it covers more values. Letting empty bins disappear. Bins that overlap (50–60 and 60–70: where does 60 go?) or leave gaps; integer division gives each value exactly one bin. Forgetting that a histogram of a small sample is bumpy by chance: with 20 values, one student more or less changes a bar by 5%.

Practice: Counting Into Equal Bins, Finished Within the Time, The Busiest Height Band

Percentages and proportions

A proportion is a part divided by a whole, a number between 0 and 1; a percentage is the same number times 100. They let you compare groups of different sizes. Twelve late buses out of 40 and 21 out of 60 are hard to compare as counts and easy as percentages:

routes = {"Route 4": (12, 40), "Route 9": (21, 60)}
for name, (late, trips) in routes.items():
    print(name, f"{100 * late / trips:.1f}% late")

Route 4 30.0% late
Route 9 35.0% late

Every percentage is a percentage of something, and most misleading percentages come from being vague about what that something is. Three traps are worth recognising.

Percentage points versus percent. If a pass rate rises from 40% to 50%, it rose by 10 percentage points, but by 25 percent (10 is a quarter of 40). Both statements are true; they answer different questions, and headlines choose whichever sounds bigger.

before, after = 40, 50
print("change in points:", after - before)
print("relative change: ", 100 * (after - before) / before, "%")

change in points: 10
relative change:  25.0 %

Small bases. "67% of our testers preferred the new design" sounds convincing until you learn it was 2 out of 3. Always report the count alongside the percentage (2 of 3, 67%), or at least the size of the group. Level 5 makes this precise with confidence intervals; for now, the rule is that a percentage of a handful of cases is closer to an anecdote than to a statistic.

Rounding that does not add up. Rounded percentages of the parts need not add to 100:

counts = {"bus": 1, "walk": 1, "bike": 1}
n = sum(counts.values())
rounded = {k: round(100 * c / n) for k, c in counts.items()}
print(rounded, "total", sum(rounded.values()))

{'bus': 33, 'walk': 33, 'bike': 33} total 99

Each group is really 33.33…%, and three rounded 33s make 99. That is not an error in the data; it is rounding. Say so in a note rather than fudging one value to force a total of 100.

Finally, a percentage change followed by the opposite percentage change does not return you to the start. A price that rises 50% and then falls 50% ends at 75% of where it began, because the second change is taken of a larger base:

price = 100
price = price * 1.5
price = price * 0.5
print(price)

75.0

Common mistakes: dividing by the wrong total (the percentage of late trips that were on Route 4 is a different number from the percentage of Route 4 trips that were late). Confusing percentage points with percent. Quoting a percentage without its count. Adding percentages from groups of different sizes: 30% of 40 trips and 35% of 60 trips are not 65% of anything; go back to the counts (33 late out of 100 trips, 33%).

Practice: Poll Results, Empty Rows Included, Points or Percent?, Which Total Do You Divide By?

Practice

These problems are required on this path at this level; the path adds one problem from each of the level's topics, picked for you and kept.

• The Ferry Delay Log
• The Middle House Price
• Clean the Log, Then Summarise
• Poll Results, Empty Rows Included
• Counting Into Equal Bins
• The Busiest Height Band

Checklist

You can now:

• Count and sum a list in one pass, including counting only the values that meet a condition.
• Compute the mean by hand, explain it as a balance point, and show how one outlier moves it.
• Combine group means correctly with a weighted mean.
• Sort to find the median for odd and even counts, and explain why it resists outliers.
• Compare mean and median to describe the shape of the data (right-skewed or roughly symmetric).
• Find all modes with a counting dictionary, and say when the mode is meaningless.
• Find the minimum, maximum and range in one pass, and use them to spot impossible values.
• Build a frequency table with equal-width bins, relative and cumulative percentages and a text histogram.
• Report percentages honestly: say what they are a percentage of, give the count, and tell percentage points from percent.