Implementing Gradient Descent for a Neural Network with One Hidden Layer

ByJu Yeon Eum

Training a neural network involves repeatedly performing three main operations:

  1. Forward propagation to calculate predictions
  2. Backpropagation to calculate gradients
  3. Gradient descent to update the parameters

This process is repeated until the cost decreases and the parameters begin to converge.

Network Architecture and Parameters

Consider a neural network with:

  • \(n^{[0]}=n_x\) input features
  • \(n^{[1]}\) hidden units
  • \(n^{[2]}\) output units
  • \(m\) training examples

For binary classification, the network normally has one output unit:\[ n^{[2]}=1 \]

A network with one hidden layer has four sets of trainable parameters:\[ W^{[1]},\quad b^{[1]},\quad W^{[2]},\quad b^{[2]} \]

Their dimensions are determined by the number of units in adjacent layers.

First-Layer Parameters

The first-layer weight matrix maps the input features to the hidden units:\[ W^{[1]} \in \mathbb{R}^{n^{[1]}\times n^{[0]}} \]

The corresponding bias is a column vector with one value for each hidden unit:\[ b^{[1]} \in \mathbb{R}^{n^{[1]}\times 1} \]

Second-Layer Parameters

The second-layer weight matrix maps the hidden-layer activations to the output units:\[ W^{[2]} \in \mathbb{R}^{n^{[2]}\times n^{[1]}} \]

Its bias vector has one value for each output unit:\[ b^{[2]} \in \mathbb{R}^{n^{[2]}\times 1} \]

For binary classification, \(n^{[2]}=1\), so:\[ W^{[2]} \in \mathbb{R}^{1\times n^{[1]}} \]

and:\[ b^{[2]} \in \mathbb{R}^{1\times 1} \]

The Cost Function

For binary classification, the network can use the same cross-entropy loss as logistic regression.

For one training example:\[ \mathcal{L} \left( \hat{y}^{(i)},y^{(i)} \right) = -\left[ y^{(i)}\log\left(\hat{y}^{(i)}\right) + \left(1-y^{(i)}\right) \log\left(1-\hat{y}^{(i)}\right) \right] \]

The cost function is the average loss across all \(m\) training examples:\[ J \left( W^{[1]},b^{[1]},W^{[2]},b^{[2]} \right) = \frac{1}{m} \sum_{i=1}^{m} \mathcal{L} \left( \hat{y}^{(i)},y^{(i)} \right) \]

The goal of training is to find parameter values that minimize \(J\).

Initializing the Parameters

Before training begins, the parameters must be initialized.

The weight matrices should be initialized with small random values rather than zeros:

W1 = np.random.randn(n1, n0) * 0.01
W2 = np.random.randn(n2, n1) * 0.01

The biases can initially be set to zero:

b1 = np.zeros((n1, 1))
b2 = np.zeros((n2, 1))

Random weight initialization is important because initializing all hidden-unit weights to zero causes the hidden units to remain symmetric. They would calculate the same outputs, receive the same gradients, and continue learning identical features.

The Gradient Descent Loop

One iteration of neural-network training consists of:

  1. Computing predictions with forward propagation
  2. Calculating the cost
  3. Computing the gradients with backpropagation
  4. Updating the parameters

Conceptually:

for iteration in range(num_iterations):
    # Forward propagation
    Z1 = W1 @ X + b1
    A1 = g1(Z1)
    Z2 = W2 @ A1 + b2
    A2 = sigmoid(Z2)
    # Compute cost
    cost = compute_cost(A2, Y)
    # Backpropagation
    dZ2 = A2 - Y
    dW2 = (1 / m) * (dZ2 @ A1.T)
    db2 = (1 / m) * np.sum(dZ2, axis=1, keepdims=True)
    dZ1 = (W2.T @ dZ2) * g1_derivative(Z1)
    dW1 = (1 / m) * (dZ1 @ X.T)
    db1 = (1 / m) * np.sum(dZ1, axis=1, keepdims=True)
    # Parameter updates
    W1 = W1 - learning_rate * dW1
    b1 = b1 - learning_rate * db1
    W2 = W2 - learning_rate * dW2
    b2 = b2 - learning_rate * db2

All of these calculations are vectorized across the complete training set.

Forward Propagation

The training examples are stored as columns of \(X\):\[ X \in \mathbb{R}^{n^{[0]}\times m} \]

The corresponding labels are stored as:\[ Y = \begin{bmatrix} y^{(1)} & y^{(2)} & \cdots & y^{(m)} \end{bmatrix} \]

For binary classification:\[ Y\in\mathbb{R}^{1\times m} \]

Forward propagation is implemented with four equations.

First-Layer Linear Calculation

\[ Z^{[1]}=W^{[1]}X+b^{[1]} \]

Its dimensions are:\[ Z^{[1]} \in \mathbb{R}^{n^{[1]}\times m} \]

Python broadcasting adds \(b^{[1]}\) to every column.

First-Layer Activation

\[ A^{[1]}=g^{[1]}(Z^{[1]}) \]

The hidden-layer activation \(g^{[1]}\) might be tanh, ReLU, or another nonlinear function.

The dimensions remain:\[ A^{[1]} \in \mathbb{R}^{n^{[1]}\times m} \]

Output-Layer Linear Calculation

\[ Z^{[2]}=W^{[2]}A^{[1]}+b^{[2]} \]

Its dimensions are:\[ Z^{[2]} \in \mathbb{R}^{n^{[2]}\times m} \]

Output-Layer Activation

\[ A^{[2]}=g^{[2]}(Z^{[2]}) \]

For binary classification, \(g^{[2]}\) is the sigmoid function:\[ A^{[2]}=\sigma(Z^{[2]}) \]

The output \(A^{[2]}\) contains all predictions:\[ A^{[2]} = \begin{bmatrix} \hat{y}^{(1)} & \hat{y}^{(2)} & \cdots & \hat{y}^{(m)} \end{bmatrix} \]

For binary classification:\[ A^{[2]} \in \mathbb{R}^{1\times m} \]

Backpropagation

The purpose of backpropagation is to compute:\[ dW^{[1]},\quad db^{[1]},\quad dW^{[2]},\quad db^{[2]} \]

Each gradient measures how the cost changes with respect to one parameter:\[ dW^{[1]} = \frac{\partial J}{\partial W^{[1]}} \]\[ db^{[1]} = \frac{\partial J}{\partial b^{[1]}} \]

and similarly for \(W^{[2]}\) and \(b^{[2]}\).

Output-Layer Error

For a sigmoid output layer with binary cross-entropy loss:\[ dZ^{[2]}=A^{[2]}-Y \]

This formula already combines the derivative of the loss with the derivative of the sigmoid function.

Because both \(A^{[2]}\) and \(Y\) have dimensions \(n^{[2]}\times m\):\[ dZ^{[2]} \in \mathbb{R}^{n^{[2]}\times m} \]

For binary classification:\[ dZ^{[2]} \in \mathbb{R}^{1\times m} \]

Output-Layer Weight Gradient

The gradient for \(W^{[2]}\) is:\[ dW^{[2]} = \frac{1}{m} dZ^{[2]}{A^{[1]}}^T \]

The matrix dimensions are:\[ \left(n^{[2]}\times m\right) \left(m\times n^{[1]}\right) = n^{[2]}\times n^{[1]} \]

Therefore:\[ dW^{[2]} \in \mathbb{R}^{n^{[2]}\times n^{[1]}} \]

This matches the dimensions of \(W^{[2]}\).

In NumPy:

dW2 = (1 / m) * (dZ2 @ A1.T)

Output-Layer Bias Gradient

The bias gradient is calculated by summing \(dZ^{[2]}\) across the training examples:\[ db^{[2]} = \frac{1}{m} \sum_{i=1}^{m} dZ^{[2](i)} \]

In NumPy:

db2 = (1 / m) * np.sum(
    dZ2,
    axis=1,
    keepdims=True
)

Setting axis=1 sums horizontally across the \(m\) training examples.

Setting keepdims=True preserves the result as a column vector:\[ db^{[2]} \in \mathbb{R}^{n^{[2]}\times 1} \]

Without keepdims=True, NumPy may return a rank-one array with shape (n2,). Rank-one arrays can behave inconsistently in later matrix operations, so preserving the two-dimensional column-vector shape is safer.

Hidden-Layer Error

The error is propagated backward from the output layer to the hidden layer:\[ dZ^{[1]} = {W^{[2]}}^T dZ^{[2]} \odot g^{[1]\prime}(Z^{[1]}) \]

The symbol \(\odot\) represents element-wise multiplication.

The first part has dimensions:\[ {W^{[2]}}^T dZ^{[2]} : \left(n^{[1]}\times n^{[2]}\right) \left(n^{[2]}\times m\right) = n^{[1]}\times m \]

The activation derivative also has dimensions:\[ g^{[1]\prime}(Z^{[1]}) \in \mathbb{R}^{n^{[1]}\times m} \]

The element-wise product is therefore valid:\[ dZ^{[1]} \in \mathbb{R}^{n^{[1]}\times m} \]

In NumPy:

dZ1 = (W2.T @ dZ2) * g1_derivative(Z1)

The @ operator performs matrix multiplication, while * performs element-wise multiplication.

Example with Tanh

If the hidden layer uses tanh:\[ A^{[1]}=\tanh(Z^{[1]}) \]

then:\[ g^{[1]\prime}(Z^{[1]}) = 1-\left(A^{[1]}\right)^2 \]

Therefore:\[ dZ^{[1]} = {W^{[2]}}^T dZ^{[2]} \odot \left( 1-\left(A^{[1]}\right)^2 \right) \]

In NumPy:

dZ1 = (W2.T @ dZ2) * (1 - A1**2)

Hidden-Layer Weight Gradient

The first-layer weight gradient is:\[ dW^{[1]} = \frac{1}{m} dZ^{[1]}X^T \]

The dimensions are:\[ \left(n^{[1]}\times m\right) \left(m\times n^{[0]}\right) = n^{[1]}\times n^{[0]} \]

Therefore:\[ dW^{[1]} \in \mathbb{R}^{n^{[1]}\times n^{[0]}} \]

This matches the dimensions of \(W^{[1]}\).

In NumPy:

dW1 = (1 / m) * (dZ1 @ X.T)

Hidden-Layer Bias Gradient

The first-layer bias gradient is:\[ db^{[1]} = \frac{1}{m} \sum_{i=1}^{m} dZ^{[1](i)} \]

In NumPy:

db1 = (1 / m) * np.sum(
    dZ1,
    axis=1,
    keepdims=True
)

The result has dimensions:\[ db^{[1]} \in \mathbb{R}^{n^{[1]}\times 1} \]

Here, preserving the dimensions is particularly important because \(db^{[1]}\) contains one bias gradient for each hidden unit.

An alternative is to sum without keepdims and then explicitly reshape the result:

db1 = (1 / m) * np.sum(dZ1, axis=1)
db1 = db1.reshape(n1, 1)

Using keepdims=True is usually simpler.

The Six Backpropagation Equations

For a two-layer binary classification network, the vectorized backward pass consists of six equations:\[ dZ^{[2]}=A^{[2]}-Y \]\[ dW^{[2]} = \frac{1}{m} dZ^{[2]}{A^{[1]}}^T \]\[ db^{[2]} = \frac{1}{m} \sum_{i=1}^{m} dZ^{[2](i)} \]\[ dZ^{[1]} = {W^{[2]}}^T dZ^{[2]} \odot g^{[1]\prime}(Z^{[1]}) \]\[ dW^{[1]} = \frac{1}{m} dZ^{[1]}X^T \]\[ db^{[1]} = \frac{1}{m} \sum_{i=1}^{m} dZ^{[1](i)} \]

Together with the four forward-propagation equations, these provide everything required to compute the gradients.

Updating the Parameters

After computing the gradients, gradient descent updates each parameter using the learning rate \(\alpha\):\[ W^{[1]} := W^{[1]}-\alpha dW^{[1]} \]\[ b^{[1]} := b^{[1]}-\alpha db^{[1]} \]\[ W^{[2]} := W^{[2]}-\alpha dW^{[2]} \]\[ b^{[2]} := b^{[2]}-\alpha db^{[2]} \]

The symbols \(=\) and \(:=\) are both commonly used to express assignment in this context. They mean that the current parameter is replaced with its updated value.

Complete Vectorized Implementation

For a tanh hidden layer and sigmoid output layer, one training iteration can be implemented as follows:

# Forward propagation
Z1 = W1 @ X + b1
A1 = np.tanh(Z1)
Z2 = W2 @ A1 + b2
A2 = 1 / (1 + np.exp(-Z2))
# Cost
cost = -(1 / m) * np.sum(
    Y * np.log(A2)
    + (1 - Y) * np.log(1 - A2)
)
# Backpropagation
dZ2 = A2 - Y
dW2 = (1 / m) * (dZ2 @ A1.T)
db2 = (1 / m) * np.sum(
    dZ2,
    axis=1,
    keepdims=True
)
dZ1 = (W2.T @ dZ2) * (1 - A1**2)
dW1 = (1 / m) * (dZ1 @ X.T)
db1 = (1 / m) * np.sum(
    dZ1,
    axis=1,
    keepdims=True
)
# Gradient descent
W1 = W1 - learning_rate * dW1
b1 = b1 - learning_rate * db1
W2 = W2 - learning_rate * dW2
b2 = b2 - learning_rate * db2

This complete process is repeated for a chosen number of iterations or until the cost converges.

Checking Gradient Dimensions

Each parameter gradient must have the same dimensions as the corresponding parameter:\[ \operatorname{shape}(dW^{[1]}) = \operatorname{shape}(W^{[1]}) = (n^{[1]},n^{[0]}) \]\[ \operatorname{shape}(db^{[1]}) = \operatorname{shape}(b^{[1]}) = (n^{[1]},1) \]\[ \operatorname{shape}(dW^{[2]}) = \operatorname{shape}(W^{[2]}) = (n^{[2]},n^{[1]}) \]\[ \operatorname{shape}(db^{[2]}) = \operatorname{shape}(b^{[2]}) = (n^{[2]},1) \]

Checking these shapes is one of the easiest ways to identify errors in a backpropagation implementation.

Key Takeaway

Training a neural network with one hidden layer requires:

  • Four equations for forward propagation
  • Six equations for backpropagation
  • Four gradient-descent parameter updates

Forward propagation computes the predictions. Backpropagation calculates how the cost changes with respect to every parameter. Gradient descent then moves each parameter in the direction that reduces the cost.

Although the full mathematical derivation relies on calculus and the chain rule, it is possible to implement the algorithm correctly by understanding the equations, tracking matrix dimensions carefully, and distinguishing matrix multiplication from element-wise multiplication.

Similar Posts

Leave a Reply