Math · Week 1 · Note 1 of 14

Functions, exponents and logarithms

Three ideas from school algebra carry most of the math in machine learning: a function is a machine, an exponent is repeated growth, and a logarithm asks “which power?”. This note rebuilds them with small numbers, then shows the one place they matter most: how a model’s score for being wrong is written.

Why this note matters

The previous note ended with a model, three houses, and a loss: one number that says how wrong the model’s guesses are. For guessing a price, the loss was the average of the squared errors. But a great deal of machine learning is not about guessing a number. It is about guessing how likely something is: that this email is spam, that the next word is “the”, that this picture shows a cat. How wrong is a guess of “70% likely” when the thing then happens? Or does not happen?

The answer that the whole field settled on is built from a logarithm, and the step that turns a model’s raw scores into percentages is built from the number $e$. Those two facts are the reason this note exists. Small things that look like notation also live here: a step size written as $3 \times 10^{-4}$, a chart with a “log scale” axis. If these are comfortable, the rest of the math track is a series of short steps.

Interviewers do not ask “what is a logarithm?”. They ask questions whose answers are built from it:

• “Why is the loss for a probability guess the negative log of the probability, not just one minus the probability?”
• “Why do we add up logs of probabilities instead of multiplying the probabilities?”
• “Your code that turns scores into percentages returns NaN on large inputs. Why, and how do you fix it?”
• “Why search step sizes on a log scale?”

By the end of this note you can answer all four, in plain words. Each is worked out below and then collected in the interview section. Words that belong to later notes are marked like this ?: hover for a one-line meaning, or ignore them on a first read.

In plain words

Each everyday idea has a precise name and symbol. You need both: the first to think, the second to speak.

Functions: machines with knobs

A function is a rule that gives exactly one output for each input. We write the rule with a name and the input in brackets: $f(x) = 2x + 1$ means “the function called $f$ takes a number $x$, doubles it and adds one”. The brackets do not mean multiplication; they mean “apply the rule to”. So $f(3) = 7$ and $f(0) = 1$.

Three things about functions matter later, so meet them now with small numbers.

1. Parameters are the knobs. $f(x) = 2x + 1$ has two numbers baked in, the 2 and the 1. Make them adjustable and write $f(x) = wx + b$. Now $w$ and $b$ are parameters: they are part of the machine, not inputs. With $w = 2, b = 1$ you get the function above; with $w = -1, b = 5$ you get a different one. When the parameters matter, we write $f(x;\, w, b)$ or $f_\theta(x)$, where $\theta$ (“theta”) is the standard name for “all the parameters together”. Training a model means: the inputs are fixed by the data, so search over $\theta$ for the knobs that give the best outputs.

2. Functions can be chained. Feed one machine’s output into another. If $g(x) = x^2$ and $f(x) = 2x + 1$, then $f(g(3))$ means “first $g$, then $f$”: $g(3) = 9$, then $f(9) = 19$. The other order gives something else: $f(3) = 7$, then $g(7) = 49$. This is composition. It matters because the big models of later weeks are nothing but long compositions: a neural network ? is “do this simple step, then this one, then this one”, dozens of times. Week 2 adds the tool for finding the slope of a composition, the chain rule.

3. A function has a picture. Plot inputs along the bottom and outputs up the side and you get its graph. $f(x) = 2x + 1$ is a straight line, which is why it is called linear. $g(x) = x^2$ is a U-shaped curve. Most of what follows is about recognising a handful of these shapes on sight; the zoo at the end collects them.

Exponents and their rules

An exponent counts repeated multiplication: $2^3 = 2 \cdot 2 \cdot 2 = 8$. The 2 is the base, the 3 is the exponent or power. The rumour after $h$ hours is $2^h$ people, and $2^{24} = 16{,}777{,}216$.

Six rules cover everything you will meet. Check each with the small numbers next to it rather than memorising it.

The negative-exponent rule is the one that shows up most. The size of each training step (the learning rate ?, week 2) is written like $3 \times 10^{-4}$, which is $3 / 10^4 = 3 / 10{,}000 = 0.0003$. In code that is 3e-4. Reading these without pausing is a small skill worth having before you tune your first model.

Two facts about Fig. 2 are used constantly. First, $a^x$ is always positive: no matter how negative $x$ gets, $e^{-x}$ only creeps towards zero and never touches or crosses it. Second, it is always increasing: a bigger $x$ always gives a bigger $e^x$, so the order of inputs is preserved in the outputs. Keep both in mind. They are exactly the two properties you need to turn a list of scores, which can be negative, into a list of percentages, which cannot, without changing which score was biggest. That recipe is below.

The number e

$e \approx 2.71828$ is a constant like $\pi$. It is the natural base for exponents because of one property, which the calculus notes will use on every page: the curve $y = e^x$ has, at every point, a slope equal to its own height. At the point where $e^x = 5$, the curve is rising at a rate of 5. No other base does this; $2^x$ rises at about 0.69 times its height, $10^x$ at about 2.3 times. Using $e$ makes every derivative involving exponentials come out without stray constants, which is the only reason mathematicians, and therefore machine learning, prefer it.

You will see $e^x$ written as $\exp(x)$ when the exponent is long, for example $\exp(-(x-\mu)^2 / 2\sigma^2)$, the formula for the bell curve you will meet in week 3. In code it is np.exp(x). A few values to have in your head:

Notice how quickly the right side explodes. $e^{10}$ is already twenty-two thousand, and $e^{710}$ is larger than the biggest number a 64-bit float can hold. A model’s raw scores can easily be in the hundreds, so “$e^x$ overflows” is a real bug you will meet, and the scores section shows the standard fix.

Logarithms: “which power?”

Formally, $\log_b x$ is the number $y$ such that $b^y = x$. The two statements say the same thing in two directions:

So $\log_2 8 = 3$ because $2^3 = 8$, and $\log_{10} 1000 = 3$ because $10^3 = 1000$. The base $b$ is which number is being raised to the power. Three bases appear in practice:

Read the shape of Fig. 3 carefully, because it is the shape of the loss for probability guesses, which the next section builds:

• $\log 1 = 0$, because $e^0 = 1$. A probability of 1 will mean zero loss.
• For inputs between 0 and 1 the log is negative, because you need a negative power of $e$ to get below 1. Probabilities live here, so log-probabilities are always negative numbers.
• Near zero the curve plunges: $\log 0.1 = -2.3$, $\log 0.01 = -4.6$, $\log 0.001 = -6.9$. Each extra zero costs another 2.3. Nothing stops the fall; the log of a number heads to minus infinity as the number heads to zero.
• Above 1 it keeps rising but ever more slowly. Going from 1 to 7.39 raises the log by 2; going from 7.39 to 54.6 raises it by 2 again. Multiplying the input by the same factor adds the same amount to the output.
• $\log 0$ and the log of any negative number do not exist. In code they come out as -inf and nan, and they are the most common source of a NaN loss.

The log rules you will use

The change-of-base rule has a practical consequence: which base you use changes nothing except a constant factor. The place where a function is smallest is the same whether you measure in $\ln$, $\log_2$ or $\log_{10}$. This is why a paper can say “log” without caring, and why the same quantity can be reported “in bits” (base 2) or “in nats” (base $e$) and mean the same thing.

Two things that look like rules but are not:

Why machine learning loves logs

Reason 1: products become sums. A model that assigns probabilities to several independent events gives the whole sequence a probability equal to the product of the individual ones. Say a model gives probabilities 0.9, 0.8 and 0.95 to three correct answers. The probability of getting all three right is

Take the log of each and add instead:

and $e^{-0.380} = 0.684$, the same answer. For three numbers this is no easier. For a million training examples, or for a 500-token sentence where each token has its own probability, a sum is something you can compute, average and take slopes of, and a product is not. Almost every model is trained by making this sum of logs as large as possible, which is the same as making the product as large as possible. In week 3 the product gets a name, the likelihood ?, and the maximum likelihood note builds on exactly this calculation.

Reason 2: tiny numbers stay representable. Multiply a thousand probabilities of 0.001 and the true answer is $10^{-3000}$. A 64-bit float cannot hold anything smaller than about $10^{-308}$, so the computer returns exactly 0, and every model you compare returns 0 too. This is called underflow. In log space the same quantity is $1000 \times \log 0.001 = -6908$, an ordinary number. The code section shows this happening.

Reason 3: confident mistakes are punished without limit. A loss should be a number that is small when the model is right and large when it is wrong, and logs of probabilities are negative, so flip the sign. The loss used for probability guesses is $-\log p$, where $p$ is the probability the model gave to the answer that turned out to be correct. It is called the log loss. Fig. 4 is its graph.

Compare this with the obvious alternative, a loss of $1 - p$. Being 90% sure and right costs 0.1 either way. But being 99% sure of the wrong answer, so $p = 0.01$ for the right one, costs 0.99 under $1 - p$ and 4.6 under $-\log p$. Push to $p = 0.0001$ and the first loss barely moves to 0.9999 while the second jumps to 9.2. The log loss keeps caring, more and more, exactly when the model is most confidently wrong, and that pressure is what teaches a model to say “70%” only when it really is right about 70% of the time. It is the forecaster’s score from the plain-words box: cheap when right, unbounded when arrogantly wrong.

A fourth, smaller reason: log scales. When you try step sizes for training, the useful candidates are 0.1, 0.01, 0.001, 0.0001: each a factor of 10 apart, not a fixed amount apart. On a log scale those are evenly spaced. The same holds for plotting a loss that falls from 1000 to 0.1: on a plain axis the interesting part is squashed into an invisible sliver at the bottom; on a log axis every factor of 10 gets equal room. Whenever a quantity varies by multiplication, think in logs.

Turning scores into percentages

For the scores $z = (2, 1, 0)$:

Check the properties: every entry is positive, they add to 1, and the biggest score got the biggest share. Written as a formula, the share for score number $i$ is

where $\sum_j$ just means “add up over all the scores”. This recipe is so common that it has a name, softmax, and the raw scores are called logits. You do not need either word yet; the softmax note in week 5 tells the full story. Here the point is only to see $e^x$ doing the job Fig. 2 promised.

Now the bug. Scores of $(1000, 1001, 1002)$ are perfectly ordinary for a model that has not been trained yet, but $e^{1000}$ is far beyond the largest float, so every term becomes inf and the division gives nan. The fix: subtract the largest score from all of them first. The exponent rule says $e^{z_i - c} = e^{z_i} / e^c$, and the $e^c$ appears in both the top and the bottom of the fraction, so it cancels. The percentages are identical; only the intermediate numbers change:

After the shift the biggest exponent is 0, so the biggest term is $e^0 = 1$ and nothing can overflow. Every machine-learning library does this inside its scores-to-percentages function, which is why hand-written versions that forget it are a classic interview question. The code section shows both versions.

The functions to recognise on sight

Most formulas in the rest of these notes are made from six shapes. If you can sketch each from memory and name where it appears, you can read the formulas as pictures instead of symbols.

Look at the sigmoid formula with fresh eyes now. It is built from Fig. 2: $e^{-x}$ is huge for very negative $x$, so $1/(1 + \text{huge})$ is nearly 0; it is nearly 0 for very positive $x$, so $1/(1 + 0)$ is nearly 1; and at $x = 0$ it is $1/(1 + 1) = 0.5$. The whole S-shape falls out of the exponential’s behaviour at the two ends.

Code

Type these out; do not paste. Each one demonstrates a sentence from above.

The rules, checked numerically

import numpy as np

a = 6.0
b = 1.5
n = 3.0

# np.log is the NATURAL log (base e). np.log2 is base 2, np.log10 is base
# 10.
# product -> sum:        2.1972... 2.1972...
print(np.log(a * b), np.log(a) + np.log(b))
# quotient -> difference: 1.3863... 1.3863...
print(np.log(a / b), np.log(a) - np.log(b))
# power -> multiple:     5.3753... 5.3753...
print(np.log(a ** n), n * np.log(a))
print(np.log(np.exp(2.5)))                     # log undoes exp:        2.5
print(np.exp(np.log(a)))                       # exp undoes log:        6.0
# change of base:        2.5850... 2.5850...
print(np.log2(a), np.log(a) / np.log(2))

# The same check written as a test: assert stops the program if the
# condition is false.
# np.isclose allows for the tiny rounding differences that decimals always
# have.
assert np.isclose(np.log(a * b), np.log(a) + np.log(b))

print(np.log(1), np.log(np.e), np.log(0.5), np.log(0.01))
# 0.0 1.0 -0.6931471805599453 -4.605170185988091

Underflow, and why log space fixes it

import numpy as np

# One thousand examples, each given probability 0.001 by a model.
p = np.full(1000, 0.001)       # an array of 1000 copies of 0.001

product = np.prod(p)           # multiply all 1000 together
log_sum = np.sum(np.log(p))    # add up their logs instead
# underflow: the true value 10^-3000 is below the smallest float
print(product)
# prints 0.0
# the same quantity in log space: an ordinary number
print(log_sum)
# prints -6907.755278982137

# A second model gives each example 0.002, so it is clearly better. Can we
# tell?
q = np.full(1000, 0.002)
# both products are 0.0
print(np.prod(q) > np.prod(p))
# prints False
# log space sees the difference
print(np.sum(np.log(q)) > np.sum(np.log(p)))
# prints True

Scores to percentages, naive and safe

import numpy as np

def scores_to_percentages_naive(z):
    # step 1: make every score positive, keep the order
    e = np.exp(z)
    # step 2: divide by the total so they add up to 1
    return e / np.sum(e)

def scores_to_percentages(z):
    # shift every score down by the biggest one (same answer, safe numbers)
    z = z - np.max(z)
    e = np.exp(z)
    return e / np.sum(e)

small = np.array([2.0, 1.0, 0.0])
big = np.array([1000.0, 1001.0, 1002.0])

print(scores_to_percentages(small).round(3))    # [0.665 0.245 0.09 ]
# exp(1000) overflows to inf; inf / inf is nan
print(scores_to_percentages_naive(big))
# prints [nan nan nan]
print(scores_to_percentages(big).round(3))      # [0.09  0.245 0.665]
# NumPy also prints "RuntimeWarning: overflow encountered in exp" for the
# naive line.
# This recipe is called softmax; the week 5 note builds on it.

The log loss

import numpy as np

def log_loss(p_correct):
    # minus the log of the probability the model gave to the right answer
    return -np.log(p_correct)

for p in [0.99, 0.9, 0.5, 0.1, 0.01]:
    loss = log_loss(p)
    # :<5 pads p to 5 characters, :.3f prints 3 decimals
    print(f"p = {p:<5}  loss = {loss:.3f}")
# p = 0.99   loss = 0.010
# p = 0.9    loss = 0.105
# p = 0.5    loss = 0.693
# p = 0.1    loss = 2.303
# p = 0.01   loss = 4.605

# Compare with the obvious alternative, 1 - p. At p = 0.01 it is only 0.99:
# it has stopped caring.
print(1 - 0.01)          # 0.99
print(log_loss(0.01))    # 4.605170185988091

Where you will meet this again

• Week 2, loss functions. The log loss, $-\log p$, becomes the standard loss for yes/no questions, with the probability $p$ coming out of the S-curve from the zoo.
• Week 3, maximum likelihood. “Add up the logs of the probabilities the model gave to what really happened, and make that as big as possible” turns out to be where almost every loss comes from. The product gets the name likelihood ?.
• Week 4, entropy. $-\log p$ averaged over outcomes measures how unpredictable something is; that is entropy ?, and the log loss gets its grown-up name, cross-entropy ?.
• Week 5, softmax. The scores-to-percentages recipe becomes softmax ?, the last step of every model that chooses between categories, and the subtract-the-max trick returns with a second trick for taking its log safely.
• Weeks 7 to 9, language models. A language model ? gives a score to every token ? it could write next, about 50,000 of them, and softmax turns those into percentages. Its training loss is $-\log p$ of the token that actually came next, averaged over every position: Fig. 4 applied billions of times. People report $e$ raised to that average loss and call it perplexity ?; a value of 10 means “as unsure as choosing among 10 equally likely tokens”. And model APIs return log-probabilities rather than probabilities, for Reasons 1 and 2 above.

The full interview answer

The most common form this note takes in an interview is “why is the loss for a probability a log?” or “why not just use one minus the probability?”. (After week 4 you will hear it asked as “why is cross-entropy a log?”; same question.) Here is a complete answer in about sixty seconds, built from the short lines above.

If they push further, be ready for:

• “Why base $e$ rather than 2?” Any base differs by a constant factor, so the minimizer is the same; $e$ makes derivatives clean. Base 2 gives the loss in bits.
• “What loss means the model was perfect?” 0: it gave probability 1 to every correct answer. In practice that is a sign of a bug or of test answers leaking into training.
• “Why does my loss show NaN?” Usually $\log 0$ somewhere: a probability that reached exactly zero, or an overflow in a hand-written scores-to-percentages step. Clip probabilities away from zero, or use the library’s loss function, which does the safe version.
• “Why are logprobs negative?” Probabilities are at most 1 and $\log 1 = 0$, so every log-probability is zero or below.

Practice questions

Answer each one out loud before opening it.

Practice problems

Do these on paper first, then check. The hand problems are the kind interviewers ask you to do on a whiteboard; the coding ones build the habits you will use every day. Solutions are hidden under each problem.

By hand

Code

import numpy as np

def check_log_rules(trials):
    # random-number generator; seed 0 repeats the same numbers
    rng = np.random.default_rng(0)
    for trial in range(trials):
        # a random number between 0.01 and 100
        a = rng.uniform(0.01, 100)
        b = rng.uniform(0.01, 100)
        n = rng.uniform(-3, 3)
        assert np.isclose(np.log(a * b), np.log(a) + np.log(b))
        assert np.isclose(np.log(a / b), np.log(a) - np.log(b))
        assert np.isclose(np.log(a ** n), n * np.log(a))
        assert np.isclose(np.log(1.0), 0.0)
        assert np.isclose(np.log(np.e), 1.0)
        assert np.isclose(np.log(np.exp(n)), n)
        assert np.isclose(np.exp(np.log(a)), a)
        assert np.isclose(np.log2(a), np.log(a) / np.log(2))
    print("all rules hold in", trials, "random trials")

check_log_rules(1000)     # all rules hold in 1000 random trials

import numpy as np

p = 0.1
direct = 1.0          # the running product
logspace = 0.0        # the running sum of logs
# the first count at which the product hits exactly 0.0
first_zero = None
for k in range(1, 401):
    direct = direct * p
    logspace = logspace + np.log(p)
    if first_zero is None and direct == 0.0:
        first_zero = k
print(first_zero)
# prints 324: 0.1 to the 324 is below the smallest positive float (about 4.9e-324)
print(logspace)
# prints -921.0340371976151: still a perfectly ordinary number
print(np.exp(logspace))    # 0.0 again, as soon as you leave log space

import numpy as np

def scores_to_percentages(z):
    z = z - np.max(z)          # subtract the biggest score
    e = np.exp(z)
    return e / np.sum(e)

def naive(z):
    e = np.exp(z)
    return e / np.sum(e)

rng = np.random.default_rng(0)
for trial in range(1000):
    # 50 random scores, some huge
    big = rng.uniform(-1000, 1000, size=50)
    p = scores_to_percentages(big)
    # nothing negative (very small shares round to 0.0)
    assert np.min(p) >= 0
    assert np.isclose(np.sum(p), 1.0)              # shares add up to 1

    # scores the naive version can handle
    small = rng.uniform(-5, 5, size=50)
    # every share agrees
    assert np.allclose(scores_to_percentages(small), naive(small))

print("safe version: never breaks, always adds to 1, matches the naive one on small scores")
print(np.sum(naive(big)))
# prints nan: the naive version on the last big scores does not even add to 1

import numpy as np
import matplotlib.pyplot as plt

# 500 evenly spaced values from 0.001 to 1
p = np.linspace(0.001, 1, 500)
plt.plot(p, -np.log(p), label="-log p")
plt.plot(p, 1 - p, label="1 - p")
# the two losses at p = 0.01
plt.scatter([0.01, 0.01], [-np.log(0.01), 0.99], color="red")
plt.xlabel("probability given to the correct answer")
plt.ylabel("loss")
plt.legend()
plt.show()
# At p = 0.01 the two losses are 4.6 and 0.99: the log loss keeps punishing
# confident mistakes
# while 1 - p has already stopped caring.

Common mistakes

All sources

Everything linked above, in the order to use it, plus one reference for later. Every np. function used on this page is explained on the NumPy reference.

1. Khan Academy, Functions (exercises). Only if function notation is rusty.
2. Khan Academy, Exponential models and Logarithms (exercises). The graded problem sets for this note.
3. 3Blue1Brown, “What’s so special about Euler’s number e?” (video, 14 min).
4. Better Explained, “An intuitive guide to exponential functions and e” and “Demystifying the natural logarithm” (articles).
5. Desmos (interactive), for the function zoo.
6. Deisenroth, Faisal and Ong, Mathematics for Machine Learning (free PDF). Not needed for this note; it is the reference for the rest of the math track, and its chapter 2 starts where the next note starts.

One-minute revision