Reference · 88 words

Glossary

Every technical word in the notes, in one plain sentence. When a note has to use a word before the note that teaches it, the word is underlined with a dotted line and a small question mark; hover over it to see the sentence below, or click it to come here.

A B C D E F G H I K L M N O P R S T V

A
accuracy: The fraction of guesses that were right. Taught properly in week 3, in the ML note “Evaluation metrics: accuracy, precision, recall, F1, ROC-AUC”.
activation function: The simple curve applied after each weighted sum, such as ReLU (zero out negatives). Without it, stacked layers would collapse into one line. Taught in What is a neural network? (week 5).
Adam: A popular optimizer: gradient descent with a memory of past steps and a separate step size per parameter. Taught properly in week 6, in the DL note “Optimizers: SGD with momentum, Adam, AdamW and the learning rate”.
attention: The part of a language model that lets each word look at the other words and decide which ones matter for guessing the next one. Taught properly in week 7, in the LLM internals note “Self-attention, the simple version”.
B
backpropagation: The chain rule applied layer by layer, from the loss backwards, to get the gradient of every parameter in a neural network. Taught properly in week 5, in the DL note “Backpropagation: how networks learn”.
baseline: The dumbest guess that could work, such as “always predict the average”. A real model must beat it. Taught in What machine learning is: data, model, loss, optimizer (week 1).
batch: A handful of examples processed together in one step, stacked as the rows of a matrix. Taught in Matrices, matrix multiplication and shapes (week 1).
BatchNorm / LayerNorm: Rescaling the numbers inside a network so each layer sees inputs on a steady range. Taught properly in week 6, in the DL note “Dropout, BatchNorm, LayerNorm and weight decay”.
Bayes’ theorem: A rule for updating how likely something is after seeing new evidence. Taught properly in week 3, in the Math note “Probability, conditional probability and Bayes’ theorem”.
bias (intercept): The parameter a model adds regardless of the input: where the line crosses the axis. Not the same as “bias” in bias–variance. Taught in Linear regression (week 1).
bias–variance trade-off: Simple models miss the pattern (bias); flexible models chase the noise (variance). You balance the two. Taught properly in week 2, in the ML note “Overfitting, underfitting and the bias–variance trade-off”.
C
calibration: A model is calibrated when its “70% sure” guesses are right about 70% of the time. Taught properly in week 3, in the ML note “Evaluation metrics: accuracy, precision, recall, F1, ROC-AUC”.
chain rule: How to find the slope of a function made of several steps: multiply the slopes of the steps. Taught properly in week 2, in the Math note “The chain rule”.
classification: Guessing a category, such as spam or not spam. Taught properly in week 3, in the ML note “Logistic regression and the sigmoid”.
context window: How many tokens the model can look at at once. Taught properly in week 10, in the Modern note “Context windows and reasoning models”.
convex: Bowl-shaped: one lowest point and no dips to get stuck in, so going downhill always reaches the bottom. Taught properly in week 2, in the ML note “Gradient descent: batch, stochastic and mini-batch”.
cosine similarity: How similar two vectors are in direction, ignoring length: 1 means same direction, 0 means unrelated, −1 means opposite. Taught in Norms, distance and cosine similarity (week 1).
cross-entropy: The general name for log loss when there are several possible answers. The loss that trains every language model. Taught properly in week 4, in the Math note “Entropy, cross-entropy and KL divergence”.
D
data leakage: Information from the test examples sneaking into training, which makes scores look better than they really are. Taught properly in week 4, in the ML note “Class imbalance, data leakage and other traps”.
derivative (slope): How fast an output changes when you nudge the input a little. The slope of the graph at that point. Taught properly in week 2, in the Math note “Derivatives: slope and rate of change”.
distribution: The full list of possible outcomes with the probability of each. Taught properly in week 3, in the Math note “Distributions, expectation and the normal distribution”.
dot product: Multiply two lists element by element and add up the results. One number that measures how much two vectors agree. Taught in Vectors and the dot product (week 1).
dropout: Randomly switching off some neurons during training so the network cannot rely on any one of them. Taught properly in week 6, in the DL note “Dropout, BatchNorm, LayerNorm and weight decay”.
E
eigenvector: A direction a matrix only stretches, without turning. The stretch factor is the eigenvalue. Taught properly in week 4, in the Math note “Eigenvectors and SVD, only as much as PCA and LoRA need”.
embedding: A list of numbers (a vector) that stands for a word, a sentence or an image, arranged so that similar things get similar lists. Taught properly in week 7, in the LLM internals note “Token embeddings: turning IDs into vectors”.
entropy: A measure of how unpredictable something is: zero when the outcome is certain, largest when every outcome is equally likely. Taught properly in week 4, in the Math note “Entropy, cross-entropy and KL divergence”.
epoch: One full pass through all the training examples. Taught properly in week 2, in the ML note “Gradient descent: batch, stochastic and mini-batch”.
example (sample, row): One data point: one house, one email, one photo. Taught in What machine learning is: data, model, loss, optimizer (week 1).
F
feature: One measured fact about an example, such as the size of a house. The inputs to the model. Taught in What machine learning is: data, model, loss, optimizer (week 1).
feature scaling: Putting all features on a similar range so that no feature dominates just because its numbers are bigger. Taught in Feature scaling, normalization and encoding (week 1).
fine-tuning: Taking a model that is already trained and training it a little more on your own data. Taught properly in week 9, in the LLM internals note “Fine-tuning for classification”.
G
generalization: Doing well on examples the model never saw during training. The real goal. Taught in What machine learning is: data, model, loss, optimizer (week 1).
gradient: The list of slopes, one per parameter, saying which way to nudge each one to raise the loss fastest. Training moves the opposite way. Taught properly in week 2, in the Math note “Partial derivatives and the gradient”.
gradient clipping: If the gradient gets too long, shrink it before taking the step, so one bad batch cannot throw training off. Taught properly in week 6, in the DL note “Optimizers: SGD with momentum, Adam, AdamW and the learning rate”.
gradient descent: Lower the loss by nudging every parameter a little in the downhill direction, over and over. Taught properly in week 2, in the ML note “Gradient descent: batch, stochastic and mini-batch”.
H
hyperparameter: A setting you choose before training, such as how big the model is or how large each training step is. Training never changes it. Taught in What machine learning is: data, model, loss, optimizer (week 1).
I
inference: Using a trained model to make guesses on new inputs. Taught in What machine learning is: data, model, loss, optimizer (week 1).
K
k-means: Split examples into k groups so that each example is close to the centre of its group. Taught properly in week 4, in the ML note “kNN and k-means: learning from similarity”.
k-nearest neighbours (kNN): Guess by looking at the k most similar examples and copying their answers. Taught properly in week 4, in the ML note “kNN and k-means: learning from similarity”.
KL divergence: A number saying how different one probability distribution is from another. Zero means identical. Taught properly in week 4, in the Math note “Entropy, cross-entropy and KL divergence”.
KV cache: Memory where a language model keeps the key and value vectors of earlier tokens so it does not recompute them for every new word. Taught properly in week 10, in the Modern note “The KV cache: why generation is fast”.
L
label (target): The right answer for an example, such as the price a house really sold for. Taught in What machine learning is: data, model, loss, optimizer (week 1).
language model (LLM): A model whose only job is to guess the next piece of text given the text so far. “Large” means billions of parameters. Taught properly in week 7, in the LLM internals note “What is an LLM? Pretraining, fine-tuning and the big picture”.
layer: A group of neurons that all read the same inputs. A network is layers in a row. Taught in What is a neural network? (week 5).
learning rate: How big each nudge is in gradient descent. Too big and training bounces around; too small and it crawls. Taught properly in week 2, in the ML note “Gradient descent: batch, stochastic and mini-batch”.
likelihood: The probability a model gives to the data that actually happened. Training picks the parameters that make it as large as possible. Taught properly in week 3, in the Math note “Maximum likelihood: where loss functions come from”.
log loss: A loss for probability guesses: minus the logarithm of the probability the model gave to the right answer. Cheap when confident and right, huge when confident and wrong. Taught properly in week 2, in the ML note “Loss functions: MSE, MAE and log loss”.
log-probability (logprob): The logarithm of a probability. Model APIs report these instead of probabilities because they are easier to add and never underflow. Taught in Functions, exponents and logarithms (week 1).
logit (raw score): A model’s raw score for one possible answer before it is turned into a probability. Can be any number, negative or huge. Taught properly in week 5, in the DL note “Logits, softmax and cross-entropy”.
LoRA: A cheap way to adapt a big model: freeze its weights and learn a small correction written as the product of two thin matrices. Taught properly in week 10, in the Modern note “LoRA, QLoRA and parameter-efficient fine-tuning”.
loss: One number that says how wrong the model’s guesses are on some examples. Zero means perfect. Training tries to make it small. Taught in What machine learning is: data, model, loss, optimizer (week 1).
M
matrix: A grid of numbers with rows and columns. A dataset is one: one row per example, one column per feature. Taught in Matrices, matrix multiplication and shapes (week 1).
mean: The average. Taught properly in week 3, in the Math note “Mean, median, variance and standard deviation”.
mean absolute error (MAE): The average size of the misses, ignoring whether each miss was too high or too low. Unlike squared error it does not punish big misses extra. Taught in What machine learning is: data, model, loss, optimizer (week 1).
mean squared error (MSE): A loss for number guesses: take each error, square it, average them. Taught in What machine learning is: data, model, loss, optimizer (week 1).
metric: The number you actually care about and report, such as accuracy. Often different from the loss the optimizer minimises. Taught in What machine learning is: data, model, loss, optimizer (week 1).
model: A recipe that turns inputs into a guess, with adjustable amounts (parameters) that training sets. Taught in What machine learning is: data, model, loss, optimizer (week 1).
N
neural network: A model built from layers, each layer multiplying by a matrix of weights and then bending the result with a simple curve. Taught in What is a neural network? (week 5).
neuron (unit): One small calculator inside a network: a weighted sum of its inputs, plus a bias, passed through a simple curve. Taught in What is a neural network? (week 5).
norm: The length of a vector. Taught in Norms, distance and cosine similarity (week 1).
normal distribution: The bell curve: most values near the middle, few far away. Taught properly in week 3, in the Math note “Distributions, expectation and the normal distribution”.
normalization: Rescaling numbers to a standard range, such as making a vector length 1 or putting a feature between 0 and 1. Taught in Feature scaling, normalization and encoding (week 1).
O
one-hot encoding: Turning a category into a list of 0s with a single 1 in the slot for that category. Taught in Feature scaling, normalization and encoding (week 1).
optimizer: The procedure that changes the parameters to lower the loss. Gradient descent is the usual one. Taught in What machine learning is: data, model, loss, optimizer (week 1).
overfitting: The model memorised its training examples, noise included, so it does well on them and badly on new ones. Taught properly in week 2, in the ML note “Overfitting, underfitting and the bias–variance trade-off”.
P
parameters (weights): The adjustable numbers inside a model. Training changes them; nothing else does. Taught in What machine learning is: data, model, loss, optimizer (week 1).
PCA: Squash many features into a few new ones that keep as much of the spread as possible. Taught properly in week 4, in the ML note “PCA and dimensionality reduction”.
perplexity: A way of reporting a language model’s loss as “how many choices it was hesitating between”. Lower is better. Taught properly in week 9, in the LLM internals note “Pretraining: the training loop, cross-entropy and perplexity”.
precision and recall: Two scores for a rare class. Precision: of the items the model flagged, how many were right. Recall: of the items that really were in the class, how many the model found. Taught properly in week 3, in the ML note “Evaluation metrics: accuracy, precision, recall, F1, ROC-AUC”.
prediction: The model’s guess for one example. Taught in What machine learning is: data, model, loss, optimizer (week 1).
probability: A number from 0 to 1 saying how likely something is. The probabilities of all the possible outcomes add up to 1. Taught properly in week 3, in the Math note “Probability, conditional probability and Bayes’ theorem”.
R
R² (R-squared): How much of the error of “always guess the average” a model removes: 1 is perfect, 0 is no better than the average, below 0 is worse. Taught in Linear regression (week 1).
RAG: Retrieval-augmented generation: look up relevant documents first, then let the language model answer using them. Taught properly in week 12, in the RAG note “The RAG pipeline end to end”.
regression: Guessing a number, such as a price. Taught in Linear regression (week 1).
regularization: Anything that discourages a model from fitting the training data too closely, usually a penalty on large parameters. Taught properly in week 3, in the ML note “Regularization: L1, L2 and early stopping”.
ReLU: The curve that keeps positive numbers and turns negatives into 0. The most common activation function. Taught in What is a neural network? (week 5).
S
sigmoid: An S-shaped curve that squashes any number into the range 0 to 1, so it can be read as a probability. Taught properly in week 3, in the ML note “Logistic regression and the sigmoid”.
softmax: The recipe that turns a list of raw scores into probabilities that add up to 1: raise e to each score, then divide by the total. Taught properly in week 5, in the DL note “Logits, softmax and cross-entropy”.
standard deviation: The square root of the variance, so it is in the same units as the data. Taught properly in week 3, in the Math note “Mean, median, variance and standard deviation”.
supervised learning: Learning from examples that come with the right answers. Taught in What machine learning is: data, model, loss, optimizer (week 1).
T
temperature: A dial at generation time: low makes the model pick its top guess almost always; high makes it take more chances. Taught properly in week 9, in the LLM internals note “Decoding: greedy, temperature, top-k and top-p”.
tensor: An array with any number of axes: a vector has one, a matrix two, a stack of images four. Taught in Matrices, matrix multiplication and shapes (week 1).
token: A piece of text, usually a word or part of a word. Language models read and write text one token at a time. Taught properly in week 7, in the LLM internals note “Tokenization and byte-pair encoding (BPE)”.
train / validation / test split: Set some examples aside before training. Train on one part, make choices on the second, measure once on the third. Taught properly in week 3, in the ML note “Train, validation, test and cross-validation”.
training: Showing a model many examples and nudging its parameters until its guesses match the answers. Taught in What machine learning is: data, model, loss, optimizer (week 1).
transformer: The kind of neural network every modern language model is built from. Its main part is attention. Taught properly in week 8, in the LLM internals note “The transformer block and the full GPT architecture”.
V
variance: How spread out numbers are: the average squared distance from the mean. Taught properly in week 3, in the Math note “Mean, median, variance and standard deviation”.
vector: An ordered list of numbers. Also an arrow from the origin to that point. Taught in Vectors and the dot product (week 1).