Neural Networks 2: Training & Backpropagation

This lecture moves from the architecture of neural networks (covered in Lecture 11) to the question of how to train them. The central topics are the forward propagation algorithm for computing predictions, the backpropagation algorithm for efficiently computing gradients, the non-convex nature of neural network optimisation, and practical techniques — including regularisation — that make training work in practice. The lecture also introduces autodifferentiation as used in modern frameworks like PyTorch.

1. Recap — Multi-Layer Neural Networks

Recall from Lecture 11 that an artificial neural network (ANN) is a nested composition of learnable linear transformations and fixed nonlinear activation functions. In compact form, a network with $L-1$ hidden layers computes:

Here, $\Theta = \{W^{(1)},\ldots,W^{(L)},\,\mathbf{b}^{(1)},\ldots,\mathbf{b}^{(L)}\}$ collects all the weight matrices and bias vectors, and $\varphi$ is applied element-wise (e.g. sigmoid, ReLU, tanh). The number of hidden layers is the depth, and the number of units in a layer is its width.

1.1 A Simple Worked Example

Consider the smallest non-trivial ANN: one input $x$, one hidden unit $v$, one output $f$, with weights $w'$ and $w$. The network computes:

The input $x$ is first scaled by $w'$, then passed through the activation $\varphi$, and finally scaled by $w$ to produce the output. This tiny network already illustrates the key pattern: linear → nonlinear → linear.

2. Forward Propagation

Forward propagation (or "forward pass") is the procedure a neural network uses to compute its output given an input $\mathbf{x}$ and a fixed set of parameters $\Theta$. It is essentially reading the nested formula from the inside out, one layer at a time.

The key observation is that each layer does the same two things: (1) compute a linear combination of the previous layer's outputs, and (2) apply the activation function. This repetitive structure is what makes the algorithm a simple loop — and is also what makes gradient computation tractable via backpropagation.

3. Training Neural Networks — The Optimisation Problem

Training a neural network means finding parameters $\Theta^*$ that minimise the empirical loss over a dataset $\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^n$:

where $\ell$ is a pointwise loss, such as the squared loss $(y - f(\mathbf{x};\Theta))^2$ for regression or the logistic loss $\log(1 + e^{-y \cdot f(\mathbf{x};\Theta)})$ for classification.

3.1 Why (Minibatch) SGD?

Despite non-convexity, we still use gradient-based methods — in particular minibatch stochastic gradient descent (SGD). There are two good reasons for this:

• Computational cost: The full gradient requires summing over all $n$ data points. With millions of parameters ($\Theta$ is very high-dimensional), this is expensive. Minibatch SGD only uses a random subset $S \subset \{1,\ldots,n\}$ per step, which is much faster.
• Escaping bad regions: The noise introduced by random sampling can actually help the optimiser escape saddle points and shallow local minima.

4. Backpropagation

The central algorithmic contribution that makes training deep networks feasible is backpropagation (short for "backward propagation of errors"). It is an efficient way to compute the gradient $\nabla_\Theta \ell$ by exploiting the layered structure of the network via the chain rule.

4.1 Intuition from the Simple Example

Return to our tiny network: $\hat{y} = w_2 \cdot \varphi(w_1 x)$, with squared loss $\ell = (y - \hat{y})^2$. We need both partial derivatives:

Notice two crucial efficiency gains: (a) the factor $2(\hat{y}-y)$ is computed once and reused, and (b) the intermediate values $z$, $h$, $\hat{y}$ were already computed during the forward pass and can be reused. Instead of recomputing everything from scratch, we propagate the error signal backward through the layers.

4.2 General Backpropagation

For an $L$-layer network, we want all gradients $\nabla_{W^{(l)}} \ell$ for $l = 1,\ldots,L$. Backpropagation computes them layer by layer, starting from the output and working backwards. At each step, only one new factor needs to be computed — everything else is reused from earlier steps or the forward pass.

4.3 Backpropagation in Matrix Form

The algorithm can be expressed concisely using an "error signal" $\boldsymbol{\delta}^{(l)}$ at each layer:

The error signal $\boldsymbol{\delta}^{(l)}$ captures "how much the loss would change if the pre-activation $\mathbf{z}^{(l)}$ changed." It flows from the output layer backward, being multiplied at each step by the transpose of the weight matrix above and the local activation derivative.

5. Autodifferentiation

In practice, nobody implements backpropagation by hand for every new architecture. Modern deep learning frameworks such as PyTorch and TensorFlow use autodifferentiation: you define the computational graph (i.e. the network architecture and loss), and the framework automatically computes all gradients for you.

Pythonmodel = torch.nn.Sequential(
    torch.nn.Linear(5, 3),   # 5 inputs → 3 hidden units
    torch.nn.ReLU(),
    torch.nn.Linear(3, 1),
    torch.nn.Flatten(0, 1)
)
loss_fn = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

for t in range(1000):
    y_pred = model(x)           # forward pass
    loss = loss_fn(y_pred, y)   # compute loss
    optimizer.zero_grad()       # reset gradients
    loss.backward()             # backpropagation
    optimizer.step()            # update parameters

6. Regularisation in Neural Networks

Neural networks often have thousands or millions of parameters, which creates a significant risk of overfitting. Several techniques help mitigate this.

6.1 Weight Decay (L2 Regularisation)

Instead of minimising just the loss, we add a penalty that keeps the weights small:

This is the same idea as ridge regression, applied to all network weights. The hyperparameter $\lambda > 0$ controls the strength of regularisation. Larger $\lambda$ means simpler models with smaller weights.

6.2 Early Stopping

During training, the training error keeps decreasing, but the validation error eventually starts to rise — this is the point where the network begins to overfit. Early stopping means halting training when the validation error stops decreasing, even if the normal convergence criterion has not been met. It is one of the simplest and most effective regularisation strategies.

6.3 Dropout

Dropout (Srivastava et al., 2014) is a regularisation technique specific to neural networks. The idea is to randomly "drop out" (i.e. set to zero) each hidden unit with probability $1-p$ during each iteration of SGD. This forces the network to not rely too heavily on any single unit and encourages redundancy across units.

During training, a different random sub-network is used at each step. At test time, all units are active, but the weights are multiplied by $p$ to compensate for the fact that more units are now present than during any single training step.

7. Key Takeaways

• Forward propagation computes the network output layer by layer: each layer applies a linear transformation followed by a nonlinear activation.
• Training a neural network is a non-convex optimisation problem, typically solved with minibatch SGD. The non-convexity means convergence depends on initialisation.
• Backpropagation efficiently computes gradients of the loss with respect to all weights by applying the chain rule layer by layer from output to input, reusing intermediate results from both the forward pass and previously computed layers.
• Autodifferentiation in frameworks like PyTorch automates backpropagation — you define the forward computation, and gradients are computed automatically.
• Regularisation (weight decay, early stopping, dropout) is essential because the large number of parameters in neural networks makes them prone to overfitting.