Math · Week 1 · Note 2 of 14

Vectors and the dot product

A vector is a list of numbers that describes one thing, and the dot product is the single operation that asks “how much do two such lists agree?”. A model’s prediction, a search for similar documents and most of what happens inside a language model are dot products. This note makes both ideas concrete with pictures and small numbers.

Why this note matters

The house-price model from the first ML note had one feature, size, and one weight. Real houses have many features: size, number of rooms, age, distance to the station. The model’s guess becomes “size times its weight, plus rooms times its weight, plus age times its weight…”. That pattern, multiply matching pairs and add them up, is the dot product, and it is the single most repeated calculation in all of machine learning. A prediction is a dot product. “Find the documents most like this question” is a dot product against every document. Inside a language model, the step that decides which earlier words matter is a dot product. If you can picture what a dot product measures, half of the diagrams you will meet later are already readable.

Questions that are really about this note:

• “Write a model’s prediction from many features as a single expression.”
• “What does the dot product between two embeddings ? tell you?”
• “What is the difference between the dot product and cosine similarity?” (Finished in the norms note.)
• “Why is the attention ? score a dot product, and what does a large value mean?” (Week 8; the dot-product half is here.)
• “What shape is this array, and why did the multiplication fail?”

In plain words

What a vector is

A vector is an ordered list of numbers. We write it in bold, $\mathbf{x}$, or with an arrow, $\vec{x}$, and its numbers with subscripts: $x_1$ is the first, $x_2$ the second, and so on up to $x_n$. The count $n$ is the vector’s dimension. $(3, 2)$ is a 2-dimensional vector; the weather report $(1, 0, 1)$ is 3-dimensional; a GPT-2 word embedding is 768-dimensional. “Dimension” here just means “how many numbers”, nothing more mysterious. Order matters: $(3, 2)$ and $(2, 3)$ are different vectors, just as “2 eggs, 1 cup flour” is a different recipe from “1 egg, 2 cups flour”.

A single plain number, like 5 or −0.3, is called a scalar, to distinguish it from a vector. The word comes from “scale”: multiplying a vector by a scalar scales it, as you will see in a moment.

Mathematicians usually write vectors standing up, as columns, and the next note explains why that convention matters once matrices arrive. In NumPy a plain vector is a one-dimensional array with shape (n,): one number in the shape means one axis. The 2-D arrow picture only works for 2 or 3 numbers, but every rule below works for 768 just the same. You cannot draw a 768-dimensional arrow, and nobody can; you compute with the list and borrow the intuition from the picture.

Adding and scaling vectors

Addition is component by component, and only between vectors of the same dimension:

Scalar multiplication multiplies every component by the same number:

A positive scalar keeps the direction and changes the length; a negative scalar also flips the arrow around. Combine the two operations and you get a linear combination: some amount of one vector plus some amount of another, $c_1\mathbf{u} + c_2\mathbf{v}$. For example $3\mathbf{u} - \mathbf{v} = (6, 3) - (1, 3) = (5, 0)$. This phrase is worth knowing because a model’s prediction is a linear combination of its features (the weights are the amounts), and “the span of these vectors” means “everything you can reach with linear combinations of them”. That is as far as span needs to go for interviews.

Subtraction is addition of a flipped vector, and it has a useful picture: $\mathbf{a} - \mathbf{b}$ is the arrow from the tip of $\mathbf{b}$ to the tip of $\mathbf{a}$. So the difference between two points is the arrow between them, which is why “distance” in the norms note will start from $\mathbf{a} - \mathbf{b}$.

The dot product

The dot product of two vectors of the same dimension multiplies matching components and sums the results:

Three things to notice. The result is a single number (a scalar), not a vector. Both vectors must have the same dimension, or there is nothing to pair up. And the order does not matter: $\mathbf{a} \cdot \mathbf{b} = \mathbf{b} \cdot \mathbf{a}$, because each pair is multiplied either way round.

Three worked examples, each used again below:

You will also see the dot product written $\mathbf{a}^\top \mathbf{b}$ (“a transpose b”) or $\langle \mathbf{a}, \mathbf{b} \rangle$. All three mean the same sum. The transpose notation comes from matrices and is explained in the next note; for now read $\mathbf{w}^\top \mathbf{x}$ as $\mathbf{w} \cdot \mathbf{x}$.

Two small but useful special cases. The dot product of a vector with a vector of all ones is its sum: $(1, 1, 1) \cdot (a, b, c) = a + b + c$. And the dot product with $(1/n, \dots, 1/n)$ is its mean. So averaging is a dot product too, which is one reason the operation shows up in so many places.

What the dot product measures

The exact statement is the single most useful formula in this note:

where $\lVert \mathbf{a} \rVert$ is the length of $\mathbf{a}$ (defined properly in the next section) and $\theta$ is the angle between the two arrows. Read it as: dot product = (length of a) × (length of b) × (how aligned they are). The cosine is the alignment dial. It is 1 when the arrows point the same way ($\theta = 0°$), 0 at a right angle ($90°$), and −1 when they point opposite ways ($180°$). You do not need any other trigonometry; those three values and “in between means in between” are enough.

Check it on the first worked example. $\mathbf{a} = (3, 4)$ and $\mathbf{b} = (4, 3)$ both have length 5 (by Pythagoras: $\sqrt{9 + 16} = 5$). Their dot product was 24, so

Two arrows of the same length, pointing almost the same way: a large dot product, as the picture promised. The second example, $(1, 2) \cdot (2, -1) = 0$, says those two arrows are at exactly 90° to each other; draw them and you will see it. Vectors with a zero dot product are called orthogonal, and the word will come back when the eigenvector note talks about directions that do not interfere with each other.

The formula also explains a subtlety that trips people up. A dot product can be large because the arrows are aligned or because one of them is simply long. Two of the word vectors from the embeddings section can have a big dot product just because one has big numbers in it. When you only care about direction, you divide the lengths out, and what remains is $\cos\theta$ itself: cosine similarity. The norms note is about exactly when to use which.

Length, unit vectors and projection

The length (also norm or magnitude) of a vector is

For $(3, 4)$ that is $\sqrt{9 + 16} = 5$; for $(1, 1, 1, 1)$ it is $\sqrt{4} = 2$. A vector of length exactly 1 is a unit vector; it carries a direction and nothing else. You make one by dividing a vector by its own length, which is called normalizing: $(3, 4) / 5 = (0.6, 0.8)$, and indeed $0.36 + 0.64 = 1$. Models that produce the word vectors of the embeddings section often normalize their outputs to unit length so that dot products between them are pure cosines.

Fig. 4 is the second way to read a dot product, and it is the one that makes trained models make sense. Drop a perpendicular from the tip of $\mathbf{a}$ onto the line through $\mathbf{b}$. The piece of that line from the origin to the foot of the perpendicular is the projection of $\mathbf{a}$ onto $\mathbf{b}$, the shadow $\mathbf{a}$ would cast if light shone straight down onto $\mathbf{b}$. Its length is

If $\mathbf{b}$ is a unit vector, the shadow is simply $\mathbf{a} \cdot \mathbf{b}$: the dot product with a unit vector measures how much of $\mathbf{a}$ lies along that direction. This is what a linear model does. Its weight vector $\mathbf{w}$ is a direction in the space of features, and $\mathbf{w} \cdot \mathbf{x}$ asks how far the input $\mathbf{x}$ extends along that direction. Training moves $\mathbf{w}$ until the direction it points in is the one that matters for the task.

One fact to file away: a dot product can never exceed the product of the two lengths, $|\mathbf{a}\cdot\mathbf{b}| \le \lVert\mathbf{a}\rVert\,\lVert\mathbf{b}\rVert$, because $|\cos\theta| \le 1$. It is called the Cauchy–Schwarz inequality, and its only job in these notes is to guarantee that cosine similarity always lies between −1 and 1.

Why the dot product is everywhere in ML

Here is the same operation wearing five different names. In each row the thing being computed is one dot product.

Why is the dot product the match score, rather than something else? Because it has the geometric meaning you just saw (big when aligned), because it is cheap (one multiply and one add per component, which GPUs do by the trillion), and because it is linear: doubling a component of $\mathbf{w}$ exactly doubles its contribution. Linearity is what makes the calculus in week 2 easy: nudge $w_i$ by a tiny amount and $\mathbf{w}\cdot\mathbf{x}$ moves by exactly $x_i$ times that amount, nothing more complicated. Hold onto that sentence; it is the reason training a linear model is simple.

Vectors that mean something: embeddings

An embedding is a vector that a model has learned to represent something: a word, a sentence, an image, a user. The defining property is that similar things get vectors that point in similar directions, so that the dot product (or cosine) between embeddings behaves like a similarity score. In Fig. 5, cat and kitten are nearly parallel; cat and car are far apart. With real embeddings the dimension is 384, 768 or more, and the axes have no human names, but the arithmetic is identical.

The famous demonstration is that vector arithmetic on word embeddings captures relationships: with the right embeddings, $\mathbf{e}_{\text{king}} - \mathbf{e}_{\text{man}} + \mathbf{e}_{\text{woman}}$ lands close to $\mathbf{e}_{\text{queen}}$. That is vector subtraction and addition from Fig. 2 doing something that looks like reasoning. It works approximately, not exactly, and only for some relationships, but it is the standard illustration that directions in embedding space can carry meaning.

This is also the entire basis of the retrieval systems in weeks 11 and 12: embed every document, embed the question, and return the documents whose vectors have the largest dot product with the question’s vector. A vector database is a system for doing that dot product against millions of vectors quickly.

Many dot products at once

Two scores on the walk inputs, written as two dot products and then as one matrix-vector product:

Each row of the matrix dotted with the column gives one entry of the result. That is all a matrix-vector product is: a stack of dot products. If you understand this note, the next one is bookkeeping about which rows pair with which columns and what shape comes out.

import numpy as np

# Two rows of weights, one row per score. A list of lists makes a 2-D array
# (a table).
W = np.array([[3.0, -2.0, -4.0],     # score 1: "should I walk?"
              [1.0, -1.0,  0.0]])    # score 2: "is the weather good?"
x = np.array([1.0, 0.0, 1.0])        # sunny, not cold, tired

# each row dotted with x: two scores in one operation
print(W @ x)
# prints [-1.  1.]
print(W.shape)        # (2, 3)   2 rows, 3 columns
print(x.shape)        # (3,)     3 numbers
print((W @ x).shape)  # (2,)     2 numbers: one score per row

Code

Type these out. Each one is a sentence from above turned into NumPy.

Vectors, shapes, addition, scaling, and the walk score

import numpy as np

x = np.array([3.0, 2.0])          # a vector: a 1-D array
print(x.shape)                    # (2,)  one number in the shape: one axis
print(x.ndim)                     # 1     the number of axes

u = np.array([2.0, 1.0])
v = np.array([1.0, 3.0])
print(u + v)                      # [3. 4.]  component by component
print(2 * v)                      # [2. 6.]  every component doubled
print(3 * u - v)                  # [5. 0.]  a linear combination

w = np.array([3.0, -2.0, -4.0])        # the walk score's weights
inputs = np.array([1.0, 0.0, 1.0])     # sunny, not cold, tired
# * pairs up components: still a vector
products = w * inputs
print(products)                        # [ 3. -0. -4.]
# add the products up: that is the dot product
print(np.sum(products))
# prints -1.0
print(np.dot(w, inputs))               # -1.0  the built-in function
# the same thing, written with @
print(w @ inputs)
# prints -1.0

The dot product three ways, the angle, and the projection

import numpy as np

def dot(a, b):
    """The definition, spelled out: multiply matching components, add them up."""
    total = 0.0
    for i in range(len(a)):
        total += a[i] * b[i]
    return total

a = np.array([3.0, 4.0])
b = np.array([4.0, 3.0])
print(dot(a, b))       # 24.0  by hand
print(np.dot(a, b))    # 24.0  NumPy's function
print(a @ b)           # 24.0  NumPy's operator

# the length is the square root of the self dot product
length_a = np.sqrt(a @ a)
length_b = np.linalg.norm(b)           # the library way to get a length
print(length_a, length_b)              # 5.0 5.0

cos_theta = (a @ b) / (length_a * length_b)
# the cosine of the angle between a and b
print(cos_theta)
# prints 0.96
angle = np.degrees(np.arccos(cos_theta))
print(round(angle, 2))                 # 16.26  the angle itself, in degrees

# the shadow (projection) of a on the direction of b
shadow = (a @ b) / length_b
print(shadow)                          # 4.8

Toy embeddings and nearest-neighbour search

import numpy as np

# Toy "embeddings": one vector per word.
# Real ones have hundreds of numbers; the arithmetic is identical.
emb = {
    "cat": np.array([0.9, 0.8, 0.1]),
    "dog": np.array([0.8, 0.9, 0.2]),
    "car": np.array([0.1, 0.2, 0.9]),
}

def similarity(word1, word2):
    return emb[word1] @ emb[word2]

print(round(similarity("cat", "dog"), 3))   # 1.46  high: the lists agree
# low: where one is big the other is small
print(round(similarity("cat", "car"), 3))
# prints 0.34

# Find the word nearest to a query vector: score every word, keep the best.
query = np.array([0.85, 0.85, 0.15])
best_word = None
best_score = None
for word in emb:
    score = emb[word] @ query
    print(word, round(score, 3))      # cat 1.46, dog 1.475, car 0.39
    if best_score is None or score > best_score:
        best_word = word
        best_score = score
print("nearest:", best_word)          # nearest: dog

Where you will meet this again

• Tomorrow, linear regression. The house model with many features is $\hat{y} = \mathbf{w}\cdot\mathbf{x} + b$, and the whole dataset’s predictions are one matrix-vector product, exactly as in the section above.
• Week 5, neural networks. A neuron ? is the walk score with a bend applied to the result; a layer ? is many of them, which is the matrix-vector product again.
• Week 7, embeddings inside a language model. Every token ? the model knows has a vector of 768 numbers in a lookup table. Feeding text to the model starts by replacing each token with its vector, so a sentence becomes a stack of vectors.
• Week 8, attention. For each pair of tokens the model computes the dot product of one token’s query vector with the other’s key vector. A large value means “pay attention to this one”. That is Fig. 3 applied to every pair of positions in the sentence. At the very end, the model’s state vector is dotted with every token’s embedding; the biggest dot product is its best guess for the next word.
• Weeks 11 and 12, retrieval. A RAG ? system turns documents and questions into vectors with an embedding model and ranks documents by dot product or cosine similarity with the question. That ranking is the “R” in RAG.

The full interview answer

The question usually arrives as “what does the dot product tell you?” or “why is similarity a dot product?”. Here is a complete answer in about seventy seconds.

If they push further, be ready for:

• “Dot product or cosine for embeddings?” Cosine if lengths are not meaningful or not normalized; dot product if the embedding model was trained for it or vectors are already unit length (then they are the same).
• “If $w_i$ changes by a small amount, how much does $\mathbf{w}\cdot\mathbf{x}$ change?” By $x_i$ times that amount. (Week 2 calls this the derivative ?, and it is why each weight’s update is proportional to the input it multiplies.)
• “Why do we scale attention ? scores by $\sqrt{d}$?” (Week 8.) Dot products of long random vectors grow with the dimension; dividing keeps them on a fixed scale. Problem 14 below shows the numbers.
• “Can two different vectors have the same dot product with a third?” Yes, infinitely many: all vectors with the same projection onto it. The dot product throws away everything perpendicular to the other vector.

Practice questions

Answer each one out loud before opening it.

Practice problems

Paper first, then check. Solutions are hidden under each problem.

By hand

Code

import numpy as np

# Plain loops only: no NumPy inside these four functions.
def dot(a, b):
    total = 0.0
    for i in range(len(a)):
        total += a[i] * b[i]
    return total

def norm(a):
    return dot(a, a) ** 0.5            # ** 0.5 is the square root

def normalize(a):
    n = norm(a)
    result = []
    for value in a:
        result.append(value / n)
    return result

def cosine_similarity(a, b):
    return dot(a, b) / (norm(a) * norm(b))

# Test against NumPy on 1000 random pairs of 50-number vectors.
rng = np.random.default_rng(0)
for trial in range(1000):
    a = rng.normal(size=50)
    b = rng.normal(size=50)
    assert np.isclose(dot(a, b), a @ b)
    assert np.isclose(norm(a), np.linalg.norm(a))
    assert np.isclose(np.linalg.norm(normalize(a)), 1.0)
    numpy_cosine = (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))
    assert np.isclose(cosine_similarity(a, b), numpy_cosine)
    # cosine always lies between -1 and 1
    assert -1.0 <= cosine_similarity(a, b) <= 1.0
print("all good")     # all good

import numpy as np

rng = np.random.default_rng(0)
for d in [2, 10, 100, 1000]:
    # 10,000 random vectors of length d, one per row
    a = rng.normal(size=(10000, d))
    b = rng.normal(size=(10000, d))
    # add across each row: one dot product per pair
    dots = np.sum(a * b, axis=1)
    mean = dots.mean()                  # the average dot product
    # how far a typical dot product is from that average
    spread = dots.std()
    ratio = spread / np.sqrt(d)
    print("d =", d, " mean =", round(mean, 2), " spread =", round(spread, 2),
          " spread / sqrt(d) =", round(ratio, 2))
# d = 2      mean = 0.01    spread = 1.42    spread / sqrt(d) = 1.01
# d = 10     mean = 0.03    spread = 3.16    spread / sqrt(d) = 1.0
# d = 100    mean = -0.03   spread = 9.93    spread / sqrt(d) = 0.99
# d = 1000   mean = 0.35    spread = 31.86   spread / sqrt(d) = 1.01
# The average stays near 0, but the spread grows like the square root of the
# dimension:
# 100 times more numbers gives about 10 times bigger dot products.

import numpy as np

rng = np.random.default_rng(1)
# 1000 "documents", one 64-number vector per row
docs = rng.normal(size=(1000, 64))
query = rng.normal(size=64)            # one query vector

def top5(scores):
    # positions, largest score first (the minus flips the order)
    order = np.argsort(-scores)
    return order[:5]                   # the first five positions

# one dot product per row: 1000 scores
dot_scores = docs @ query
# the length of every row
lengths = np.linalg.norm(docs, axis=1)
cos_scores = dot_scores / (lengths * np.linalg.norm(query))
print("dot   ", top5(dot_scores))      # [834 471 525 942 916]
print("cosine", top5(cos_scores))      # [834 471 126 525 236]

# make document 7's vector ten times longer
docs[7] = docs[7] * 10
dot_scores = docs @ query
lengths = np.linalg.norm(docs, axis=1)
cos_scores = dot_scores / (lengths * np.linalg.norm(query))
# document 7 jumps to the top
print("dot after   ", top5(dot_scores))
# prints [  7 834 471 525 942]
# unchanged: ignores length
print("cosine after", top5(cos_scores))
# prints [834 471 126 525 236]

import numpy as np
import matplotlib.pyplot as plt

a = np.array([2.0, 1.0])
# 361 angles all the way round the circle
angles = np.linspace(0, 2 * np.pi, 361)
# b = (cos angle, sin angle) is a unit vector pointing at that angle, so a .
# b is:
dots = a[0] * np.cos(angles) + a[1] * np.sin(angles)
degrees = np.degrees(angles)

plt.plot(degrees, dots)
plt.axhline(0, color="gray")                  # the zero line
# True where the dot product is (almost) zero
near_zero = np.abs(dots) < 0.02
plt.scatter(degrees[near_zero], dots[near_zero], color="red")
plt.xlabel("angle of b (degrees)")
plt.ylabel("a . b")
plt.title("a . b for unit vectors b: a cosine wave of height sqrt(5)")
plt.show()
# where b is perpendicular to a
print(degrees[near_zero].round(1))
# prints [117. 297.]

Common mistakes

All sources

Everything linked above, in the order to use it, plus one reference.

1. 3Blue1Brown, “Vectors | Chapter 1, Essence of linear algebra” (video, 10 min). Start here.
2. 3Blue1Brown, “Linear combinations, span, and basis vectors | Chapter 2” (video, 10 min).
3. 3Blue1Brown, “Dot products and duality | Chapter 9” (video, 14 min; first half).
4. Khan Academy, Vectors (Precalculus) and Vectors and spaces (Linear algebra) (exercises). The graded problem sets.
5. Better Explained, “Understanding the dot product” (article).
6. Jay Alammar, “The Illustrated Word2vec” (article, first half) and the TensorFlow Embedding Projector (interactive).
7. NumPy, numpy.dot (documentation).
8. Deisenroth, Faisal and Ong, Mathematics for Machine Learning, sections 2.1–2.2 and 3.1–3.3 (free PDF). The formal version, for after the note has settled.

One-minute revision