DL · Week 5 · Note 1 of 8

What is a neural network?

A neural network is a big adjustable formula. This note builds it up from its smallest part, the neuron, to a full network, with worked numbers at every step.

Why this note matters

Everything later in these notes is built from the idea on this page. A large language model such as GPT is a neural network. Attention ?, embeddings ?, fine-tuning ? and LoRA ? are all ways of arranging or adjusting the same basic part: a layer of numbers that multiplies its input and bends the result. Everything you need to read this note was covered in weeks 1 to 4: the dot product, matrices and shapes, the loss, gradient descent and the chain rule.

Interviewers rarely ask “what is a neural network?” to an experienced candidate, but they constantly ask questions that depend on it: why activation functions are needed, how many parameters a layer has, what shape a tensor is after a layer, why weights cannot start at zero. If this page is solid, those are easy marks.

Before reading on, watch this. It gives you the pictures; this note gives you the words and the detail.

In plain words

Each everyday idea has an exact technical name. You need both: the first to understand, the second to speak.

A worked example

Let us put real numbers on the walk decision. Each input is 1 for yes and 0 for no. Suppose these are the weights and the bias:

The rule: add everything up, and go if the total is above zero.

Day A: sunny, not cold, not tired.

Day B: sunny, cold and tired.

That is a complete neuron. Notice three things. A positive weight makes an input argue for “yes”, a negative weight makes it argue for “no”, and the size of the weight is how loudly it argues. The bias moves the bar: with a bias of +5 you would go on almost any day; with −5 almost never.

Here we chose the weights by hand. The whole point of machine learning is that we do not. We show the network many days along with what the right decision was, and an algorithm adjusts the weights until the outputs match. How that adjustment works is the subject of the next two notes.

One neuron, precisely

A neuron does two steps.

1. Weighted sum. Multiply every input by its weight, add the results, add the bias. The total is the pre-activation, written $z$. Mathematically this is a dot product of the weight vector and the input vector, plus the bias.
2. Activation. Pass $z$ through a fixed function $f$. The result $a$ is the neuron’s output, and becomes an input to neurons in the next layer.

In the walk example the activation was a hard rule: output 1 if $z > 0$, otherwise 0. That is called a step function, and a neuron that uses it is the original perceptron, invented by Frank Rosenblatt in 1958. Modern networks replace the hard step with smoother functions, for a reason that matters a lot: training works by nudging weights and watching whether the output gets slightly better. A step function is flat everywhere, so a small nudge changes nothing and gives no signal about which direction to move. Smooth activations have a slope, and the slope is the signal.

What one neuron can and cannot do

To draw this, use just two yes/no inputs, so that every possible situation fits on a flat chart. Put the first input along the bottom and the second up the side. There are only four situations: (0,0), (0,1), (1,0) and (1,1). Those are the four dots in each half of Fig. 2.

A decision one neuron can make: “both must be yes”. Say you only go walking when it is sunny and you have a friend free. Give both inputs a weight of 1 and set the bias to −1.5:

It works. Now look at where the total is exactly zero: every point with $x_1 + x_2 = 1.5$. On the chart those points form one straight line, the red line on the left of Fig. 2. Everything above and to the right of that line totals more than zero (go); everything below and to the left totals less (stay). The neuron has drawn a line and said “this side yes, that side no”.

Now the names for what you just saw:

• This logic rule, “yes only when both are yes”, is called AND. Programmers know it as the && operator.
• The dividing line is the decision boundary. The weights set its tilt, and the bias slides it without tilting it. Try bias −0.5 instead of −1.5 and the line moves so that three dots are “go”: that is the OR rule.
• Data that one straight line can split correctly is called linearly separable. “Linear” here just means “straight”.
• With three inputs the chart is 3-D and the boundary becomes a flat sheet, called a plane. With more inputs you cannot draw it, but the idea is identical, and the general name is a hyperplane. “Hyper” means nothing scary: it is “the same flat cut, in more dimensions”.

A decision one neuron can never make: “exactly one of the two”. Picture a staircase light with a switch at the top and a switch at the bottom. The light is on when exactly one switch is up. Both down: off. Both up: off. This rule is called XOR, short for “exclusive or”.

Look at the right half of Fig. 2. The two “on” dots sit on opposite corners, (0,1) and (1,0), and the two “off” dots sit on the other two corners. Try to draw one straight line with both filled dots on one side and both empty dots on the other. You cannot. The dashed lines show two attempts; each one puts a dot on the wrong side.

You do not have to trust the picture. Here is the same thing with arithmetic. Suppose some weights $w_1, w_2$ and bias $b$ did solve XOR. Then four things would have to be true at once:

1. Both off, (0,0): the total is just $b$, and it must be at or below zero.
2. Only the second on, (0,1): $w_2 + b$ must be above zero.
3. Only the first on, (1,0): $w_1 + b$ must be above zero.
4. Both on, (1,1): $w_1 + w_2 + b$ must be at or below zero.

Add lines 2 and 3 together: $w_1 + w_2 + 2b$ is above zero. But line 4 says $w_1 + w_2 + b$ is at or below zero, and line 1 says $b$ is at or below zero, so their sum $w_1 + w_2 + 2b$ is also at or below zero. The same number cannot be both. So no weights exist. One straight cut cannot do “exactly one”.

This is not a trivia point. In 1969 Marvin Minsky and Seymour Papert made exactly this argument in their book Perceptrons, and interest in neural networks collapsed for years. The way out, which you will see worked through with numbers below, is to use more than one neuron, arranged in layers, with a bend between the layers. The next two sections explain what that bend is and why it is needed.

Why the bend matters

Take two tiny neurons with one input each and no activation function, so each one just outputs its total. The first has weight 2 and bias 1. The second reads the first one’s output, with weight 3 and bias −4:

Now feed the first into the second and simplify:

The chain of two neurons is exactly the same as one neuron with weight 6 and bias −1. Check it: for $x = 2$, the first neuron gives 5, the second gives 11, and the single neuron gives $6 \cdot 2 - 1 = 11$. Two layers of “multiply and add” always squash down into one layer of “multiply and add”. Add a third layer and it squashes again. A hundred such layers are still one straight cut, and still cannot do XOR.

Here is the name for this. A rule where the output changes in steady proportion to the input, so that its graph is a straight line, is called linear. The weighted sum is linear. Chaining linear rules gives a linear rule, which is the squashing you just saw. Anything that bends the graph is non-linear, and the activation function is where the network gets its bend. Once each layer bends its output, the next layer combines bent pieces instead of straight ones, and bent pieces can be put together into any shape.

For whole layers, where each neuron has many inputs, the same substitution is written with the weight tables you will meet in the shapes section. You can skip past this line for now and come back after that section; it says nothing the number example did not:

Read it as: “layer two applied to layer one” equals “one layer with weights $W_2W_1$ and bias $W_2\mathbf{b}_1 + \mathbf{b}_2$”. Same squash, bigger numbers. This is the single most common “why” question about neural networks, so learn the small example well enough to write it on a whiteboard.

The activation functions to know

Read the three pictures left to right:

• ReLU (“rectified linear unit”, said “rell-you”) is the simplest ramp: if the total is negative, output 0; if it is positive, pass it through unchanged. In the walk example: “a bad day counts as zero, a good day counts as exactly how good it is”. Its formula $\max(0, z)$ means “the bigger of 0 and $z$”.
• Sigmoid is an S-shaped ramp that squeezes any total into a number between 0 and 1. A total of 0 gives 0.5, a big positive total gives nearly 1, a big negative total gives nearly 0. Because the output looks like a probability (“70% sure it is a yes”), it is used when the network must answer a yes/no question with a confidence.
• GELU is ReLU with the corner rounded off. For clearly positive totals it is the same as ReLU; near zero it dips slightly below zero and curves smoothly instead of bending sharply. It is what GPT uses.

You should be able to name each function in the table below, sketch its shape, and state one strength and one weakness. The “slope” column matters because training multiplies slopes together across layers; the backpropagation note shows why.

Three words in that table come up in interviews, so here is what each means in plain terms:

• Saturation. Look at the sigmoid picture: for totals beyond about ±4 the curve goes flat. A neuron out there is “saturated”: nudging its total changes its output by almost nothing, so training gets almost no signal from it.
• Vanishing gradients. Training passes the signal backwards through every layer, multiplying slopes as it goes. Sigmoid’s slope is at most 0.25, so after ten layers the signal is at most $0.25^{10}$, which is about one millionth. The early layers barely learn. ReLU’s slope of exactly 1 on the positive side is why it replaced sigmoid inside networks.
• Dying ReLU. ReLU’s flat side has slope 0. If a neuron’s weights drift so that its total is negative for every input, its output is always 0 and its slope is always 0, so no training signal ever reaches it again. It is dead. GELU and SiLU avoid this by keeping a tiny slope on the negative side.

In the GELU formula, $\Phi(z)$ is the chance that a value from a standard bell curve lands below $z$. For a big positive $z$ that chance is nearly 1, so GELU returns nearly $z$, like ReLU. For a big negative $z$ it is nearly 0, so GELU returns nearly 0, also like ReLU. In between it blends smoothly. That is the whole trick: ReLU without the sharp corner.

One more activation deserves a mention here, although it gets its own note later this week: softmax, the scores-to-percentages recipe from the logarithms note. It is used on the output layer when the network must choose among many options, and it turns a list of scores into percentages that add up to 100%. A language model’s final step is a softmax over every token in its vocabulary: “the next token is ‘the’ with 31%, ‘a’ with 12%, …”.

From neuron to network

Now the vocabulary, one word at a time:

• A row of neurons that all read the same inputs is a layer.
• When every neuron in a layer is connected to every output of the layer before it, the layer is fully connected. You will also see it called a “dense” layer or, in PyTorch, a “linear” layer. All three mean the same thing.
• A stack of fully connected layers is a multilayer perceptron (MLP), also called a feed-forward network. “Feed-forward” means the numbers flow one way, from input to output, with no loops back.

Fig. 4 has three columns of circles. Each column has a name:

• The input layer on the left is just the facts about the day, written as a list of numbers. A list of numbers is called a vector. This layer does no computing and has nothing to learn; it is only where the data enters.
• Hidden layers are the rows in the middle. They are called “hidden” because their values are neither the input you provide nor the answer you read out. You never look at them directly.
• The output layer on the right gives the answer, for example one score per possible choice.

Two more words. The number of neurons in a layer is its width (Fig. 4’s hidden layer has width 4). The number of layers is the network’s depth. “Deep learning” just means networks with many layers.

Pushing one day’s numbers through the network from left to right, layer by layer, is called the forward pass. It is what happens every time you use a trained model, which is called inference, and it is also the first half of every training step.

What do the hidden neurons end up asking? Nobody assigns them a question. During training each one drifts towards detecting whatever pattern helps reduce the error. In a network that looks at photos, first-layer neurons usually end up responding to edges, later ones to textures and shapes, and the last ones to whole objects such as “a cat’s ear”. Learning these in-between descriptions automatically is called representation learning. Before deep learning, people wrote those descriptions by hand, and it was the slowest, hardest part of the job. This is the main reason deep learning took over.

A hidden layer solves XOR

Here is that story as a tiny network: two inputs, two hidden neurons using ReLU (negative becomes 0), and one output neuron with no activation:

So helper 1 has weights 1 and 1 with bias 0. Helper 2 has weights 1 and 1 with bias −1. The boss has weights 1 and −2. Run all four inputs through it by hand:

All four rows are right. Look at the first row: helper 2’s total is −1, and ReLU turns that into 0. That is the bend doing its job. Remove the ReLU and helper 2 would give −1, 0, 0, 1 down the column, the output column would become 2, 1, 1, 0, and the network would be back to being one straight cut that gets the first row wrong.

There is a second way to see what happened, and it is the way to think about every hidden layer from now on. The hidden layer re-described each day using two new numbers, $(h_1, h_2)$, instead of the original $(x_1, x_2)$. In the original numbers, the four dots could not be split by a line. In the new numbers they can: the boss is a single neuron, which means a single straight cut, and it works. A hidden layer turns a problem that is not linearly separable into one that is. That is the job of every hidden layer: re-describe the data so that the next layer’s job becomes easy.

Matrices and shapes

Start small. Suppose a hidden layer has two neurons, both reading the three walk inputs (sunny, cold, tired). Neuron 1 is the walk neuron from the worked example, weights (3, −2, −4). Neuron 2 asks “is the weather good?” with weights (1, −1, 0). Stack the two rows into a table and put the inputs in a column:

The input column says: sunny, not cold, tired. “Multiply the table by the column” means: take each row, multiply it against the column item by item, and add up. Row 1 gives $3\cdot1 + (-2)\cdot0 + (-4)\cdot1 = -1$; row 2 gives $1\cdot1 + (-1)\cdot0 + 0\cdot1 = 1$. Add the biases and you get the two totals $(-2, 1)$. Row 1 is exactly the walk neuron’s calculation from before; row 2 is the second neuron’s. One table multiplication did both.

Written generally, a whole layer is this one line, where $f$ is the activation applied to each total separately:

Now the shapes. For the hidden layer of Fig. 4, with 3 inputs and 4 neurons:

The table has one row per neuron (4) and one column per input (3), so its shape is “4 by 3”. In code that is written (out_features, in_features): outputs first, inputs second. The bias has one number per neuron, and so does the result. Applying the activation to each of the four totals separately is called applying it element-wise.

One more layer of bookkeeping, and then you are done. In practice a network never processes one day at a time. It takes a stack of days, say 32 of them, and processes all 32 together. That stack is a batch. The input then becomes a table with one row per day and one column per input, so its shape is (batch_size, in_features), here 32 by 3. The layer turns it into a table of shape (batch_size, out_features), 32 by 4: one row of four answers for each day.

The rule to remember: a linear layer only ever changes the last number in the shape, the one that counts features. Everything in front of it, like the batch, passes through untouched. You will meet this again in a transformer, where the input shape is (batch, sequence_length, embedding_size), three numbers instead of two, and a linear layer still changes only the last one.

This is also why deep learning needs GPUs. The work is almost entirely multiplying big tables of numbers, which breaks into thousands of small, independent multiply-and-add steps. A GPU is a chip built to do thousands of small steps at the same time.

Counting parameters

Count Fig. 4 by hand. Every line between two circles is one weight. The hidden layer has 4 neurons each reading 3 inputs, which is 3 × 4 = 12 lines, plus one bias per neuron, 4 more: 16 numbers. The output layer has 2 neurons each reading 4 hidden values, 4 × 2 = 8 lines, plus 2 biases: 10 numbers. The activation functions have no adjustable numbers at all; ReLU is the same rule forever.

In general a fully connected layer with $n$ inputs and $m$ neurons has $m \times n$ weights and $m$ biases. Same arithmetic, bigger numbers. GPT-2 small has 124 million parameters, counted exactly this way across all its layers. The count shows up in interviews as “how big is this layer?”, “how much memory does this model need?” and, later, “how many parameters does LoRA train?”.

The memory question has a one-line answer. Each parameter stored as a standard 32-bit number takes 4 bytes, so a model’s size in memory is roughly 4 bytes × the parameter count. For 124 million parameters that is about 0.5 GB. Store each number in 16 bits instead and it is half that. Your laptop can hold GPT-2 small easily; a 70-billion parameter model at 16 bits needs 140 GB, which is why it does not fit on one consumer GPU.

Do not confuse parameters with hyperparameters. Parameters are the numbers the network learns from data. Hyperparameters are the choices you make before training starts: how many layers, how wide, which activation, how big a step to take when nudging weights. A handy test: if training changes it, it is a parameter; if you set it in the config, it is a hyperparameter.

Why deep, not just wide

The famous result is the universal approximation theorem. It says: a network with just one hidden layer and a bending activation can get as close as you like to any continuous function, over any limited range of inputs, provided the hidden layer is wide enough. “Continuous” means no sudden jumps, and “as close as you like” means you pick the allowed error and the theorem promises a width that achieves it.

That sounds as if depth is unnecessary. It is not, because of what the theorem leaves out:

• It does not say how many neurons “wide enough” is. For many functions the number is impractically huge.
• It says that good weights exist. It does not say that training will ever find them.
• It says nothing about doing well on new data the network has not seen, which is the thing you actually want.

Depth fixes the first problem. A deep network builds features in a hierarchy, reusing simple features to make complex ones, the way a few hundred words are reused to build millions of sentences. For some functions a one-row network needs exponentially more neurons than a deep one to do the same job. In practice deep networks also train to better answers on structured data such as images and language. That is the empirical reason: people tried both, and deep won.

How the weights start

Now the names. Setting the starting values is initialization. The twin problem is why you never start all weights at the same value, including zero. Using random starts to avoid it is called symmetry breaking: “symmetry” here just means “all the neurons look the same”.

The whisper-and-shout problem is handled by picking the size of the random numbers from the layer’s size. A neuron with 1,000 inputs adds up 1,000 products, so its weights must be smaller than those of a neuron with 10 inputs, or its totals will be far bigger. Two standard recipes do this arithmetic for you:

• Xavier (Glorot) initialization, designed for sigmoid and tanh. Paper: Glorot and Bengio, 2010.
• He (Kaiming) initialization, designed for ReLU, which zeroes half its inputs and so needs weights about 1.4 times larger to compensate. Paper: He et al., 2015.

You do not need the formulas yet; PyTorch applies a sensible default when you create a layer. What you need is the idea: random to break the twins, scaled to the layer size to keep the signal steady. Biases are normally started at zero, which is fine because the random weights already make the neurons different. GPT-2 starts its weights from a bell curve centred on zero with a spread (standard deviation) of 0.02.

Code

Type these out yourself rather than pasting. Before running each one, write down what you expect it to print.

One neuron in NumPy

This is Fig. 1 with numbers: a dot product, a bias, and ReLU.

import numpy as np

x = np.array([1.0, 2.0, 3.0])        # three inputs
w = np.array([0.5, -1.0, 2.0])       # one weight per input
b = 0.1                              # bias

# weighted sum: 0.5 - 2.0 + 6.0 + 0.1 = 4.6
z = np.dot(w, x) + b
a = max(0.0, z)                      # ReLU: keep z if positive, else 0
print(z, a)                          # 4.6 4.6

The XOR network in NumPy

The same network as the hand-worked table above, now in matrix form. X @ W1.T is the matrix multiplication that computes every neuron’s weighted sum for all four inputs at once. The transpose .T is needed because each row of W1 is one neuron’s weights and each row of X is one example.

import numpy as np

# All four inputs, one per row. dtype=float stores them as decimals.
X = np.array([[0, 0],
              [0, 1],
              [1, 0],
              [1, 1]], dtype=float)

W1 = np.array([[1, 1],      # row 1: weights of hidden neuron h1
               [1, 1]])     # row 2: weights of hidden neuron h2
b1 = np.array([0, -1])      # biases of h1 and h2
W2 = np.array([[1, -2]])    # output neuron: y = 1*h1 - 2*h2
b2 = np.array([0])          # bias of the output neuron

# weighted sums of both hidden neurons, for all 4 inputs: shape (4, 2)
S = X @ W1.T + b1
H = np.maximum(0, S)        # ReLU on every entry
Y = H @ W2.T + b2           # output neuron for all 4 inputs: shape (4, 1)

print(H)
# [[0. 0.]
#  [1. 0.]
#  [1. 0.]
#  [2. 1.]]
print(Y.ravel())            # [0. 1. 1. 0.]  -> exactly XOR

Optional preview: the network of Fig. 4 in PyTorch

import torch
from torch import nn

model = nn.Sequential(
    # 3 inputs -> 4 hidden neurons: 3*4 weights + 4 biases = 16
    nn.Linear(3, 4),
    nn.ReLU(),         # the bend; has no parameters
    # 4 hidden -> 2 outputs:        4*2 weights + 2 biases = 10
    nn.Linear(4, 2),
)

x = torch.tensor([[1.0, 2.0, 3.0]])   # shape (1, 3): a batch of one example
y = model(x)                          # the forward pass
print(y.shape)                        # torch.Size([1, 2])

first_layer = model[0]                # the first nn.Linear
# -> (out_features, in_features)
print(first_layer.weight.shape)
# prints torch.Size([4, 3])
print(first_layer.bias.shape)         # torch.Size([4])

# numbers in each weight and bias tensor
counts = [p.numel() for p in model.parameters()]
print(counts)                         # [12, 4, 8, 2]
print(sum(counts))                    # 26

nn.Linear(3, 4) stores a weight matrix of shape (4, 3) and a bias of shape (4,), and computes $\mathbf{x}W^\top + \mathbf{b}$. The weights start as small random numbers, so the output is meaningless until the network is trained.

Where you will meet this again

None of this is just history. Every transformer ? block inside GPT contains a two-layer feed-forward network of exactly the kind on this page: a linear layer, a GELU activation, and another linear layer. In GPT-2 small the first of those layers goes from 768 numbers to 3,072, which is $768 \times 3072 + 3072 = 2{,}362{,}368$ parameters in one layer.

Within each transformer block, these feed-forward layers hold roughly two thirds of the weights, and attention ? holds the other third. So when you study the GPT architecture in the LLM internals track, a large part of each block will already be familiar. The Vizuara course builds this block in code, and you can follow along from the start of the playlist:

The full interview answer

The short blue lines above are what to say for each sub-topic. This is how they join into one answer to “explain what a neural network is”. Aim for about ninety seconds.

If they push further, be ready for:

• “Show me why it collapses.” Write the two-layer substitution from this section.
• “How many parameters?” Inputs × outputs + outputs, per layer.
• “Why not initialize to zero?” Symmetry: identical neurons get identical updates.
• “Why ReLU over sigmoid?” Sigmoid’s slope is at most 0.25 and it saturates, so gradients vanish in deep networks.

Practice questions

Answer each one out loud before opening it.

Common mistakes

All sources

Everything linked above, in the order to use it, plus two deeper references.

1. 3Blue1Brown, “But what is a neural network?” (video). Start here.
2. StatQuest, “The Essential Main Ideas of Neural Networks” (video). A second, slower explanation if the first did not click.
3. Michael Nielsen, Neural Networks and Deep Learning, chapter 1 and chapter 4 (free online book).
4. TensorFlow Playground (interactive).
5. Andrej Karpathy, micrograd (video, code).
6. Sebastian Raschka, Build a Large Language Model (From Scratch), Appendix A, with its code; chapter 4 for the feed-forward block and GELU.
7. Goodfellow, Bengio and Courville, Deep Learning, chapter 6: “Deep Feedforward Networks” (free online). The rigorous version, including the XOR example.
8. Universal approximation theorem (Wikipedia), for the precise statements and their history.
9. Glorot, Bordes and Bengio, “Deep Sparse Rectifier Neural Networks”, 2011. An influential paper showing ReLU works well in deep networks.

One-minute revision