ML · Week 1 · Note 1 of 15

What machine learning is: data, model, loss, optimizer

Machine learning is programming by example: instead of writing the rules, you show the computer data and let it find rules that fit. Every ML system, from a line through three points to GPT, has the same four parts. This note names them, builds the smallest possible one by hand, and gives you the vocabulary for the rest of the track.

Why this note matters

Interviewers for AI roles ask the big-picture question more often than juniors expect, because it reveals whether you have a mental model or a pile of memorized terms. “Explain machine learning to someone non-technical.” “What is the difference between supervised and unsupervised learning?” “What are the parts of a training loop?” “What is a hyperparameter?” These are easy marks if you have one clear picture, and that picture is the four-part loop on this page. Every later note, including the ones on transformers, is a specialization of it.

Questions that are really this note:

• “How is machine learning different from ordinary programming?”
• “Walk me through what happens when a model trains.”
• “Parameters versus hyperparameters?”
• “Is training a language model supervised or unsupervised?”
• “Why do we split data into train and test sets?”

In plain words

The four parts of every ML system

Take the parts one at a time, with the example this track will use for the next two weeks: predicting a house’s price from its size.

Data. A set of examples, each with features and, for supervised learning, a label. Here each example is one house: the feature is its size, the label is the price it sold for. The data is usually written as a matrix $X$ with one row per example (the data matrix from the matrices note) and a vector $\mathbf{y}$ of labels. Data is the part people underestimate: in practice, more and cleaner data beats a cleverer model far more often than the other way round.

Model. A function with adjustable parameters that turns features into a prediction. The simplest one for house prices is a line, $\hat{y} = w \cdot \text{size} + b$, with two parameters. A neural network ? (week 5) is the same idea with millions of parameters and a more flexible shape. Choosing the form of the model (line? tree? network? how big?) is your job; choosing the parameter values is the optimizer’s.

Loss. A single number that says how wrong the model’s predictions are on the data, with zero meaning perfect. For prices, the standard choice is the mean squared error: take each prediction minus its label, square it so that over- and under-predictions both count as bad, and average. The loss turns the vague goal “predict well” into a precise target: make this number small.

Optimizer. The procedure that changes the parameters to reduce the loss. The one everyone uses is gradient descent: compute which way each parameter should move to reduce the loss by a little, move all of them that way, repeat. Week 2 is about exactly how. For now the key insight is that the optimizer never sees the problem; it only sees the loss number and the knobs, and it turns knobs to push the number down.

Run the loop: predict on the data, compute the loss, update the parameters, predict again. Each pass through the whole dataset is an epoch. Stop when the loss stops improving, or when a clock runs out. The parameters at the end are the trained model; “saving a model” means saving those numbers.

A first model, by hand

Write size in thousands of square feet, so the data is $x = (1, 1.5, 2)$ and the prices are $y = (200, 300, 380)$. The model is a line with two knobs:

The loss is the mean squared error over the three houses:

Now play optimizer. Try three settings of the knobs and score each one:

The third setting is best of the three, and you found it the way every optimizer does: by looking at how wrong the current guess was and moving the knobs in the direction that reduces the error. The second row was too high for the big house, so you lowered the slope a little and lifted the intercept to keep the small houses right. Gradient descent will do exactly this reasoning with calculus instead of intuition, and the code section does it with brute force. The best line of all turns out to be $w = 180,\ b \approx 23.3$, with a loss of about 22.2; you will derive that in the linear regression note.

Notice what “learning” meant here. Nothing mystical: a model shape, a score, and a search over the knobs. The model never “understood” houses. It found two numbers that make a score small. Keep this deflationary picture; it is accurate all the way up to the largest language model, and it is what interviewers want to hear.

The real goal: new data

The ability to do well on unseen data is called generalization, and it is the actual objective of machine learning. The training loss is only a proxy for it. A model that drives training loss to zero by memorizing the examples, like the wiggly curve, has overfit: it learned the noise along with the pattern. A model too simple to capture the pattern at all has underfit. The whole craft of ML sits between those two failures, and the overfitting note next week gives it a full treatment.

The practical rule follows immediately. Before you train anything, split the data: a training set the model learns from and a test set it never sees until the very end, when you measure it once. Most projects add a third validation set for making choices during development, so that the test set stays untouched. Any number you report from data the model trained on is not evidence of anything. The data splits note in week 3 has the details and the traps.

Kinds of learning

Two clarifications that interviewers like. First, regression and classification differ only in the type of the label, and that changes the loss: squared error for numbers, and a different loss for categories that you meet next week (the log loss ?). The rest of the loop is identical. Second, a word you will hear constantly: a language model is a model whose only job is to guess the next piece of text given the text so far, and the pieces of text it works with, usually words or parts of words, are called tokens. “Large language model” (LLM) just means one with billions of parameters. So the question “is training an LLM supervised?” has a precise answer: the mechanics are supervised (there is a label and a loss at every position), but the labels cost nothing because they are the next tokens of the text itself. That is what “self-supervised” means, and it is why LLMs could be trained on the whole internet without anyone labelling it.

The workflow around the loop

1. Define the task and the metric. What is the input, what is the output, and what single number will tell you the model is good enough? The metric is what the business cares about (accuracy, revenue, time saved) and is often different from the loss the optimizer minimizes. Decide it first, or you will decide it after seeing results, which is cheating.
2. Get the data and split it. Training, validation and test sets, separated before you look too closely. Clean it, and look at examples by hand; most project-ending bugs live in the data.
3. Build a baseline. The simplest thing that could work: predict the average, predict the most common class, or use a rule of thumb. Every real model must beat the baseline on the validation set, and the margin by which it does so is the honest measure of what you have gained. The next section works this out in full.
4. Train. The loop of Fig. 1.
5. Evaluate on held-out data. The validation set during development. The test set once, at the end, for the number you report.
6. Iterate. Change one thing at a time: a feature, a model size, a hyperparameter. Measure again. Keep a log; the log is where the interview stories come from.
7. Deploy and monitor. The world changes after you ship, and so does the data. A model that was good in March can be bad in September. Monitoring is part of the job, and the production track in week 15 is about it.

This is also the structure of Project 0 and of every project after it. Projects that follow this shape produce numbers you can defend; projects that skip the baseline and the split produce numbers nobody should believe, including you.

Baselines: is the model any good?

A baseline is the simplest predictor that could reasonably be used, usually one that ignores the features entirely, against which every real model is compared. It turns a raw number (“MSE 3424”) into a meaningful one (“a third less error than guessing the average”). It also catches bugs: a model that does worse than the baseline is broken, not just weak.

The three houses, by hand

Go back to the three houses: sizes 1, 1.5 and 2 (thousand square feet), prices 200, 300 and 380. The baseline ignores the sizes and guesses the average price for every house:

The best line, which does use the size, had a mean squared error of 22.2. Now the number means something: using the size removed almost all of the error the lazy guess made.

One number for “how much better”

Compare the two errors with one ratio. The model’s error divided by the baseline’s error says what fraction of the lazy guess’s error is left. One minus that is the fraction the model removed:

Read it as “the model removes 99.6% of the error you would make by guessing the average”. Three things to know:

• 1 (100%) means a perfect model. 0 means no better than the baseline.
• Below 0 means worse than guessing the average. That is a bug or a badly broken model.
• When the error is mean squared error and the baseline is the mean, this number has a famous name, R² ? (“R-squared”). You will meet it properly in the linear regression note. It is nothing more than “improvement over the mean baseline”.

Why the average, and not some other number?

If you must guess one number for everyone, which number is best depends on how mistakes are scored. There are two common ways. Mean squared error squares each miss and averages, as before. Mean absolute error (MAE) just averages the sizes of the misses, ignoring whether each was too high or too low: misses of −20 and +10 give (20 + 10) / 2 = 15. Try four guesses on the three house prices, scored both ways:

Squared error punishes big misses heavily, so the best single guess is pulled toward the far-away prices: that is the mean. Absolute error treats every unit of miss the same, so the best single guess is the middle value: the median. So the right baseline matches the metric: mean for squared error, median for absolute error. This is also why a single huge price in the data drags a mean-based model around while a median barely moves.

The usual baselines

Always compare against the strongest simple baseline, not the weakest. Guessing zero for every patient is technically a baseline, and any model beats it by miles, which proves nothing. The code below shows exactly this trap.

Code: the three houses

import numpy as np

# the three houses from earlier
prices = np.array([200.0, 300.0, 380.0])

# The baseline ignores the sizes and guesses the same price for every house.
guess = prices.mean()                            # 293.33
baseline = np.full_like(prices, guess)           # [293.33 293.33 293.33]
baseline_mse = np.mean((baseline - prices) ** 2)
print(round(guess, 2), round(baseline_mse, 1))   # 293.33 5422.2

# The best line from earlier uses the sizes.
sizes = np.array([1.0, 1.5, 2.0])
model_mse = np.mean((180 * sizes + 23.33 - prices) ** 2)
print(round(model_mse, 1))                       # 22.2

# How much of the baseline's error did the sizes remove?
skill = 1 - model_mse / baseline_mse
print(f"{skill:.1%}")                            # 99.6%

# Why the mean? Try other constant guesses and score each one two ways.
for c in [250, prices.mean(), np.median(prices), 350]:
    mse = np.mean((c - prices) ** 2)             # mean squared error
    mae = np.mean(np.abs(c - prices))            # mean absolute error
    print(f"guess {c:6.1f}   MSE {mse:7.1f}   MAE {mae:5.1f}")
# guess  250.0   MSE  7300.0   MAE  76.7
# guess  293.3   MSE  5422.2   MAE  62.2    <- the mean wins on squared error
# guess  300.0   MSE  5466.7   MAE  60.0    <- the median wins on absolute error
# guess  350.0   MSE  8633.3   MAE  76.7

Code: a real dataset, three baselines and a model

The diabetes dataset has 442 patients. For each one there are 10 measurements (age, body mass index, blood pressure and six blood tests) and a score for how much the disease progressed one year later, between 25 and 346. The code splits off 89 patients as a test set, scores three baselines and a linear model on them, and reports the skill.

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, mean_absolute_error

# 442 patients, 10 measurements each; y = disease progression score (25 to 346)
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0)

# Step 1: the baselines. Each is ONE number, computed from the TRAINING
# answers only, then guessed for every one of the 89 test patients.
guesses = {"mean": y_train.mean(), "median": np.median(y_train), "zero": 0.0}
for name, value in guesses.items():
    pred = np.full_like(y_test, value)
    mse = mean_squared_error(y_test, pred)
    mae = mean_absolute_error(y_test, pred)
    print(f"{name:>6} baseline   guess {value:6.1f}   MSE {mse:6.0f}   MAE {mae:5.1f}")

# Step 2: the real model, which reads the 10 measurements.
model = LinearRegression().fit(X_train, y_train)
pred = model.predict(X_test)
model_mse = mean_squared_error(y_test, pred)
model_mae = mean_absolute_error(y_test, pred)
print(f"linear regression                MSE {model_mse:6.0f}   MAE {model_mae:5.1f}")

# Step 3: report the model against the strongest honest baseline.
baseline = np.full_like(y_test, y_train.mean())
baseline_mse = mean_squared_error(y_test, baseline)
skill = 1 - model_mse / baseline_mse
print(f"skill over the mean baseline: {skill:.0%}")

# The square root of an MSE is back in the score's own units: a typical
# miss, in points.
baseline_miss = np.sqrt(baseline_mse)
model_miss = np.sqrt(model_mse)
print(f"typical miss: baseline {baseline_miss:.0f} points, model {model_miss:.0f} points")

# Output:
#   mean baseline   guess  151.6   MSE   5135   MAE  59.2
# median baseline   guess  139.0   MSE   5360   MAE  59.0
#   zero baseline   guess    0.0   MSE  28913   MAE 154.2
# linear regression                MSE   3424   MAE  46.2
# skill over the mean baseline: 33%
# typical miss: baseline 72 points, model 59 points

How to read the output:

• The mean baseline is the bar to clear. Guessing 151.6 for everyone gives a mean squared error of 5,135, a typical miss of about 72 points.
• The median wins on absolute error and loses on squared error, exactly as in the house table. The theory shows up even on real data.
• The zero baseline is the trap. Against an error of 28,913, the model looks like a miracle. It is not; zero is just a silly guess.
• The model removes about a third of the baseline’s error (skill 33%), cutting the typical miss from about 72 to about 59 points. That is real but modest, and saying so honestly is what an interviewer wants to hear.
• The baseline’s guess comes from the training answers (y_train.mean()), never the test answers. Using the test answers would let the baseline peek at the exam, the same cheating the test set exists to prevent.

scikit-learn ships the lazy guess as a model of its own, so it slots into any pipeline:

from sklearn.dummy import DummyRegressor
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0)

# A baseline packaged as a model: same fit / predict as every other model.
# strategy can be "mean", "median" or "constant".
dummy = DummyRegressor(strategy="mean").fit(X_train, y_train)
# same as by hand
print(mean_squared_error(y_test, dummy.predict(X_test)))
# prints 5134.78...

# For categories: DummyClassifier(strategy="most_frequent")
# always predicts the majority class.

Code: when the baseline is already “95% accurate”

For categories the trap is sharper. When one class is rare, always guessing the common one already scores a high accuracy while doing nothing useful:

import numpy as np

# 1,000 emails, about 5% of them spam (1 = spam, 0 = not spam).
rng = np.random.default_rng(0)
# 1,000 random numbers between 0 and 1
draws = rng.random(1000)
# True for about 5% of them, then True/False -> 1/0
is_spam = (draws < 0.05).astype(int)
print(is_spam.sum(), "spam emails")                       # 45 spam emails

# The majority-class baseline: always say "not spam".
baseline = np.zeros_like(is_spam)
# True where the guess matches the truth
right = baseline == is_spam
# True counts as 1, False as 0
accuracy = np.mean(right)
# sounds great
print(f"accuracy {accuracy:.1%}")
# prints 95.5%
# & means "both true", email by email
caught = np.sum((baseline == 1) & (is_spam == 1))
# catches nothing
print("spam caught:", caught)
# prints 0

# So a spam filter with 95% accuracy may have learned nothing at all.
# On lopsided data, compare against this baseline, and use a metric
# that looks at the rare class.

So “our spam filter is 95% accurate” means nothing until you know the majority-class baseline is also about 95%. Week 3 covers the metrics that look at the rare class directly, called precision and recall ?.

The vocabulary, in one table

These are the words that will appear on every page from here on. Most confusion in interviews comes from using two of them interchangeably when they are not, so the right-hand column gives the pairs to keep apart.

AI, ML, deep learning, LLMs

The reason to be precise about the boxes is that interviewers use them to probe depth. “Deep learning” is not a different kind of learning; it is machine learning with a particular, very flexible model and the same four parts. “Classical” machine learning, the subject of this track, refers to the models that are not deep networks: linear models, trees, nearest neighbours, clustering. They still win on most tabular business data, they train in seconds on a laptop, and they are what interviewers reach for when they want to check that you understand the fundamentals rather than only the fashion.

Code

Type these out. The three programs are the same loop at three levels of automation.

The four parts, by brute force

import numpy as np

# Three houses: size in thousands of square feet, price in thousands.
sizes = np.array([1.0, 1.5, 2.0])
prices = np.array([200.0, 300.0, 380.0])

def predict(w, b, x):              # the MODEL: a line with two knobs
    return w * x + b

# the LOSS: mean squared error over the data
def loss(w, b):
    errors = predict(w, b, sizes) - prices
    return np.mean(errors ** 2)

print(loss(180, 0), loss(200, 0), loss(190, 10))
# 566.666... 133.333... 41.666...   (matches the table)

# The dumbest OPTIMIZER: try every knob setting on a grid and keep the best
# one.
best_loss = None
for w in range(150, 251, 2):       # slopes 150, 152, ..., 250
    for b in range(-50, 51, 2):    # intercepts -50, -48, ..., 50
        current = loss(w, b)
        if best_loss is None or current < best_loss:
            best_loss = current
            best_w = w
            best_b = b
# -> price ~ 180 * size + 24 on this coarse grid
print(best_w, best_b, round(best_loss, 2))
# prints 180 24 22.67

# That is machine learning in miniature: data, a model with knobs, a loss,
# and a search over the knobs.
# Gradient descent (week 2) replaces the grid with a smarter search;
# everything else stays the same.

The same model with scikit-learn

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# scikit-learn wants 2-D: one row per house, one column per feature
sizes = np.array([[1.0], [1.5], [2.0]])
prices = np.array([200.0, 300.0, 380.0])

# the model: y = w x + b, knobs not yet set
model = LinearRegression()
# the optimizer: finds the best w and b for the data
model.fit(sizes, prices)
print(model.coef_)                # [180.]   the slope w the model learned
print(round(model.intercept_, 2))  # 23.33    the intercept b: the exact best line; the grid got close

new_house = np.array([[1.8]])     # a house of 1,800 sq ft we have not seen
# inference: the predicted price
print(model.predict(new_house).round(2))
# prints [347.33]

predictions = model.predict(sizes)
# the loss at the best knobs
print(round(mean_squared_error(prices, predictions), 2))
# prints 22.22

A real dataset, a split and a baseline

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# 442 patients, 10 features, target = disease progression
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0)
# (89, 10)   rows, columns of each part
print(X_train.shape, X_test.shape)
# prints (353, 10)

# Baseline first: always predict the training mean. Any real model must beat
# this (see Baselines above).
baseline = np.full_like(y_test, y_train.mean())
baseline_mse = mean_squared_error(y_test, baseline)
print("baseline test MSE:", round(baseline_mse))     # 5135

# make the model and train it in one line
model = LinearRegression().fit(X_train, y_train)
train_mse = mean_squared_error(y_train, model.predict(X_train))
test_mse = mean_squared_error(y_test, model.predict(X_test))
print("train MSE:", round(train_mse))                # 2735
print("test  MSE:", round(test_mse))                 # 3424

# The test number is the one that matters: it estimates performance on
# patients we have never seen.
# It is a little worse than train, which is normal. If it were much worse,
# we would suspect overfitting.

Where you will meet this again

• Data. About 40 GB of web text. Each training example is a window of 1,024 tokens, and the label at each position is simply the next token. No human labelled anything: self-supervised.
• Model. A particular kind of neural network called a transformer ?, with 124 million parameters (small) up to 1.5 billion (large). A function from a token sequence to a probability for each possible next token, with knobs.
• Loss. The log loss ?: minus the logarithm of the probability the model gave to the token that actually came next, averaged over all positions. The next note builds it; week 4 gives it its grown-up name, cross-entropy ?.
• Optimizer. A polished variant of gradient descent called Adam ? (week 6), run for hundreds of thousands of steps, each on a batch of sequences.
• Hyperparameters. Number of layers, width, learning rate, batch size, how many tokens the model sees at once. Chosen by people, before training, and reported in the paper.
• Inference. Run the trained model on a prompt, get a probability for each next token, pick one, append it, repeat. That is “generation”.
• Generalization. Measured as loss on held-out text the model never trained on; a model that memorized its training set would have low training loss and useless outputs on anything new, exactly the wiggly curve of Fig. 3.

Fine-tuning, RAG and agents, the subjects of the later tracks, are ways of adapting or wrapping this one trained function. Keep locating the four parts and none of it will feel like magic.

The full interview answer

The question usually comes as “explain machine learning to me as if I were not technical, then as if I were”. Here is both, in about ninety seconds.

If they push further, be ready for:

• “Give an example of a hyperparameter and how you would choose it.” Learning rate; try several values, pick the one with the best validation loss, never the test loss.
• “Loss versus metric?” Loss is what you can optimize (smooth); the metric is what you report (accuracy, F1). They should agree in direction, but they are different numbers.
• “Why not evaluate on the training data?” It measures memorization, not learning. The wiggly curve has zero training error and is useless.
• “Is deep learning a different kind of ML?” No; same four parts with a neural network as the model.
• “What would you do first on a new problem?” Define the metric, split the data, build the dumbest baseline, then try the simplest model that could work.

Practice questions

Answer each one out loud before opening it.

Practice problems

Paper first, then check. Solutions are hidden under each problem.

By hand

Code

import numpy as np
import matplotlib.pyplot as plt

sizes = np.array([1.0, 1.5, 2.0])
prices = np.array([200.0, 300.0, 380.0])

def loss(w, b):
    return np.mean((w * sizes + b - prices) ** 2)

# A finer grid: steps of 0.5 instead of 2.
ws = np.arange(150, 251, 0.5)      # 150.0, 150.5, ..., 250.0
bs = np.arange(-50, 51, 0.5)       # -50.0, -49.5, ..., 50.0
best_loss = None
for w in ws:
    for b in bs:
        current = loss(w, b)
        if best_loss is None or current < best_loss:
            best_loss = current
            best_w = w
            best_b = b
# (exact optimum: b = 23.33, MSE 22.22)
print(best_w, best_b, round(best_loss, 2))
# prints 180.0 23.5 22.25

# The loss as a function of w alone, with b held at its best value.
losses = []
for w in ws:
    losses.append(loss(w, 23.33))
plt.plot(ws, losses)
plt.xlabel("w (with b = 23.33)")
plt.ylabel("MSE")
plt.title("a bowl: one minimum, no traps")
plt.show()
# The loss of a linear model under MSE is a parabola in each parameter (a
# bowl in both).
# That is why gradient descent works so cleanly for it: roll downhill and
# you reach the bottom.

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0)

guesses = {"mean": y_train.mean(), "median": np.median(y_train), "zero": 0.0}
for name, value in guesses.items():
    # the same guess for every test patient
    baseline = np.full_like(y_test, value)
    error = mean_squared_error(y_test, baseline)
    print(f"{name:>6} baseline MSE: {error:8.0f}")

model = LinearRegression().fit(X_train, y_train)
model_mse = mean_squared_error(y_test, model.predict(X_test))
print(f"linear regression MSE: {model_mse:8.0f}")

mean_baseline = np.full_like(y_test, y_train.mean())
mean_mse = mean_squared_error(y_test, mean_baseline)
improvement = 1 - model_mse / mean_mse
print(f"improvement over mean baseline: {improvement:.0%}")
#   mean baseline MSE:     5135
# median baseline MSE:     5360
#   zero baseline MSE:    28913
# linear regression MSE:     3424
# improvement over mean baseline: 33%
# "The gap" is the distance from the baseline's error down to a perfect 0. The model closes a third of it.
# Always report the improvement over the baseline, not the raw number: raw
# MSE means nothing on its own.

import numpy as np

# 30 noisy points around the straight line y = 0.8 x + 1.
rng = np.random.default_rng(0)
# 30 random x values between 0 and 10, in order
x = np.sort(rng.uniform(0, 10, 30))
# random wobble, typically between -1 and +1
noise = rng.normal(0, 1, 30)
y = 0.8 * x + 1 + noise

# Every second point for training, the points in between for testing.
x_train = x[::2]
y_train = y[::2]
x_test = x[1::2]
y_test = y[1::2]
print(len(x_train), len(x_test))             # 15 15

for degree in [1, 3, 12]:
    # fit a curve of this degree to the training points
    coeffs = np.polyfit(x_train, y_train, degree)
    train_mse = np.mean((np.polyval(coeffs, x_train) - y_train) ** 2)
    test_mse = np.mean((np.polyval(coeffs, x_test) - y_test) ** 2)
    print(f"degree {degree:>2}: train MSE {train_mse:6.2f}   test MSE {test_mse:8.2f}")
# degree  1: train MSE   0.94   test MSE     0.89    the straight line fits the pattern
# degree  3: train MSE   0.62   test MSE     0.86    a gentle curve: about the same on test
# degree 12: train MSE   0.15   test MSE     4.74    hugs the training points, five times worse on test:
# it memorized the noise (Fig. 3, left panel)

import numpy as np

def train_val_test_split(X, train=0.7, val=0.15, seed=0):
    # The test share is whatever is left: 1 - 0.7 - 0.15 = 0.15.
    n = len(X)
    rng = np.random.default_rng(seed)
    # the row numbers 0..n-1 in a random order
    order = rng.permutation(n)
    n_train = int(n * train)
    n_val = int(n * val)
    # the first 70% of the shuffled order
    train_rows = order[:n_train]
    val_rows = order[n_train:n_train + n_val]       # the next 15%
    test_rows = order[n_train + n_val:]             # the rest
    return train_rows, val_rows, test_rows

X = np.arange(200).reshape(100, 2)      # 100 rows, 2 features
train_rows, val_rows, test_rows = train_val_test_split(X)
X_train = X[train_rows]                 # pick those rows out of X
print(len(train_rows), len(val_rows), len(test_rows), X_train.shape)
# prints 70 15 15 (70, 2)

# Self-checks: no row is in two parts, and together the parts cover every
# row.
# & on two sets keeps what they share
assert len(set(train_rows) & set(val_rows)) == 0
assert len(set(train_rows) & set(test_rows)) == 0
assert len(set(val_rows) & set(test_rows)) == 0
all_rows = np.concatenate([train_rows, val_rows, test_rows])
assert sorted(all_rows) == list(range(100))
print("all checks passed")              # all checks passed
# Shuffling before cutting matters: if the rows were sorted by label, an
# unshuffled split would be useless.

Common mistakes

All sources

Everything linked above, in the order to use it, plus the two books for the track.

1. StatQuest, “A Gentle Introduction to Machine Learning” (video, 12 min). Start here.
2. Andrew Ng, Supervised Machine Learning: Regression and Classification (Coursera, course 1 of the Machine Learning Specialization; free to audit). Week 1 of the course matches this note and the next two.
3. Google ML Crash Course, Generalization and the ML Glossary.
4. Kaggle Learn, Intro to Machine Learning (hands-on, about 3 hours).
5. scikit-learn, LinearRegression and load_diabetes (documentation).
6. Raschka, Machine Learning with PyTorch and Scikit-Learn, chapter 2 code. Chapters 1–2 of the book are the reading for this note.
7. James, Witten, Hastie and Tibshirani, An Introduction to Statistical Learning (free PDF), chapter 2: “Statistical Learning”. The careful version of generalization, with the bias-variance picture you will meet next week.

One-minute revision