Computing the Output of a Neural Network

A neural network with one hidden layer can be understood as several logistic-regression-like units working together. Each unit performs two basic operations:

1. A linear calculation
2. A nonlinear activation

The hidden layer performs these calculations across multiple units, and the output layer repeats the same general process to produce the final prediction.

From Logistic Regression to a Neural Network

For logistic regression, the prediction is computed in two steps:\[ z=w^Tx+b \]\[ a=\sigma(z) \]

where the sigmoid function is:\[ \sigma(z)=\frac{1}{1+e^{-z}} \]

The activation \(a\) is also the prediction:\[ a=\hat{y} \]

A neural network applies this same pattern multiple times. Each hidden unit has its own weights and bias, allowing different units to learn different functions from the same input.

Neural Network Notation

Consider a neural network with:

• Three input features
• Four hidden units
• One output unit

The notation:\[ a_i^{[l]} \]

contains two indices:

• The superscript \([l]\) identifies the layer.
• The subscript \(i\) identifies a unit within that layer.

For example:\[ a_1^{[1]} \]

is the activation of the first unit in the first hidden layer.

Square brackets indicate the layer number. This is different from parentheses, which are used to identify individual data examples:\[ x^{(i)} \]

Computation in the First Hidden Unit

The first hidden unit computes:\[ z_1^{[1]} = \left(w_1^{[1]}\right)^T x+b_1^{[1]} \]

It then applies the sigmoid function:\[ a_1^{[1]} = \sigma\left(z_1^{[1]}\right) \]

This is structurally identical to logistic regression, but the parameters belong specifically to the first unit of the hidden layer.

Computation in the Other Hidden Units

The second hidden unit computes:\[ z_2^{[1]} = \left(w_2^{[1]}\right)^T x+b_2^{[1]} \]\[ a_2^{[1]} = \sigma\left(z_2^{[1]}\right) \]

Similarly, the third and fourth hidden units compute:\[ z_3^{[1]} = \left(w_3^{[1]}\right)^T x+b_3^{[1]} \]\[ a_3^{[1]} = \sigma\left(z_3^{[1]}\right) \]

and:\[ z_4^{[1]} = \left(w_4^{[1]}\right)^T x+b_4^{[1]} \]\[ a_4^{[1]} = \sigma\left(z_4^{[1]}\right) \]

Each unit receives the same input vector \(x\), but it uses a different weight vector and bias.

Why a Loop Is Unnecessary

The four hidden units could be calculated separately with a loop, but that would be inefficient. Instead, their parameters and outputs can be stacked into matrices and vectors.

This produces a compact vectorized implementation of the entire hidden layer.

Constructing the Hidden-Layer Weight Matrix

Stack the transposed weight vectors vertically:\[ W^{[1]} = \begin{bmatrix} \left(w_1^{[1]}\right)^T\\ \left(w_2^{[1]}\right)^T\\ \left(w_3^{[1]}\right)^T\\ \left(w_4^{[1]}\right)^T \end{bmatrix} \]

If the input contains three features, each transposed weight vector has shape:\[ (1,3) \]

Stacking four of them produces:\[ W^{[1]}\in\mathbb{R}^{4\times3} \]

Each row of \(W^{[1]}\) contains the weights for one hidden unit.

Constructing the Bias Vector

The four hidden-unit biases are stacked vertically:\[ b^{[1]} = \begin{bmatrix} b_1^{[1]}\\ b_2^{[1]}\\ b_3^{[1]}\\ b_4^{[1]} \end{bmatrix} \]

Therefore:\[ b^{[1]}\in\mathbb{R}^{4\times1} \]

Vectorizing the Linear Computation

The input vector is:\[ x= \begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix} \]

with shape:\[ x\in\mathbb{R}^{3\times1} \]

The complete hidden-layer linear calculation is:\[ z^{[1]}=W^{[1]}x+b^{[1]} \]

Expanding the matrix multiplication gives:\[ W^{[1]}x = \begin{bmatrix} \left(w_1^{[1]}\right)^Tx\\ \left(w_2^{[1]}\right)^Tx\\ \left(w_3^{[1]}\right)^Tx\\ \left(w_4^{[1]}\right)^Tx \end{bmatrix} \]

After adding the bias vector:\[ z^{[1]} = \begin{bmatrix} \left(w_1^{[1]}\right)^Tx+b_1^{[1]}\\ \left(w_2^{[1]}\right)^Tx+b_2^{[1]}\\ \left(w_3^{[1]}\right)^Tx+b_3^{[1]}\\ \left(w_4^{[1]}\right)^Tx+b_4^{[1]} \end{bmatrix} \]

This is exactly:\[ z^{[1]} = \begin{bmatrix} z_1^{[1]}\\ z_2^{[1]}\\ z_3^{[1]}\\ z_4^{[1]} \end{bmatrix} \]

The values for different units are stacked vertically.

Computing the Hidden-Layer Activations

The activation vector is defined as:\[ a^{[1]} = \begin{bmatrix} a_1^{[1]}\\ a_2^{[1]}\\ a_3^{[1]}\\ a_4^{[1]} \end{bmatrix} \]

Apply the sigmoid function element-wise:\[ a^{[1]}=\sigma\left(z^{[1]}\right) \]

This means:\[ a^{[1]} = \begin{bmatrix} \sigma\left(z_1^{[1]}\right)\\ \sigma\left(z_2^{[1]}\right)\\ \sigma\left(z_3^{[1]}\right)\\ \sigma\left(z_4^{[1]}\right) \end{bmatrix} \]

The two equations for the hidden layer are therefore:\[ \boxed{z^{[1]}=W^{[1]}x+b^{[1]}} \]\[ \boxed{a^{[1]}=\sigma\left(z^{[1]}\right)} \]

Treating the Input as Layer Zero

The input vector can also be written as the activation of layer zero:\[ a^{[0]}=x \]

Using this notation, the first hidden layer becomes:\[ z^{[1]}=W^{[1]}a^{[0]}+b^{[1]} \]\[ a^{[1]}=\sigma\left(z^{[1]}\right) \]

This notation reveals a pattern that extends naturally to deeper neural networks.

Checking the Hidden-Layer Dimensions

For this example:\[ W^{[1]}:(4,3) \]\[ a^{[0]}=x:(3,1) \]\[ b^{[1]}:(4,1) \]

Therefore:\[ z^{[1]} = W^{[1]}a^{[0]}+b^{[1]} \]

has dimensions:\[ (4,3)(3,1)+(4,1)=(4,1) \]

The activation function does not change the dimensions:\[ a^{[1]}:(4,1) \]

This makes sense because the hidden layer contains four units, so it produces four activation values.

Computation in the Output Layer

The output layer uses the hidden-layer activations as its input:\[ z^{[2]}=W^{[2]}a^{[1]}+b^{[2]} \]

The final activation is:\[ a^{[2]}=\sigma\left(z^{[2]}\right) \]

Because the network has one output unit:\[ W^{[2]}\in\mathbb{R}^{1\times4} \]\[ a^{[1]}\in\mathbb{R}^{4\times1} \]\[ b^{[2]}\in\mathbb{R}^{1\times1} \]

Thus:\[ z^{[2]}:(1,1) \]

and:\[ a^{[2]}:(1,1) \]

The final output is the network’s prediction:\[ \boxed{\hat{y}=a^{[2]}} \]

The Four Forward-Propagation Equations

The complete forward computation for a neural network with one hidden layer is:\[ \boxed{ z^{[1]}=W^{[1]}a^{[0]}+b^{[1]} } \]\[ \boxed{ a^{[1]}=\sigma\left(z^{[1]}\right) } \]\[ \boxed{ z^{[2]}=W^{[2]}a^{[1]}+b^{[2]} } \]\[ \boxed{ a^{[2]}=\sigma\left(z^{[2]}\right)=\hat{y} } \]

In NumPy-style code:

Z1 = np.dot(W1, X) + b1
A1 = sigmoid(Z1)
Z2 = np.dot(W2, A1) + b2
A2 = sigmoid(Z2)

For a single input example, X can be treated as a column vector.

Dimension Summary

Extending the Method to Multiple Examples

The equations above describe a single input example. To process multiple examples efficiently, place each input vector in a separate column of a matrix:\[ X= \begin{bmatrix} | & | & & |\\ x^{(1)} & x^{(2)} & \cdots & x^{(m)}\\ | & | & & | \end{bmatrix} \]

The same four equations can then compute the outputs for all examples simultaneously, with broadcasting used to add the bias vectors to each column.

Key Takeaway

A neural network with one hidden layer repeatedly applies the same two-step pattern used by logistic regression:\[ \text{linear calculation} \rightarrow \text{activation function} \]

For the entire network:\[ z^{[1]}=W^{[1]}a^{[0]}+b^{[1]} \]\[ a^{[1]}=\sigma\left(z^{[1]}\right) \]\[ z^{[2]}=W^{[2]}a^{[1]}+b^{[2]} \]\[ a^{[2]}=\sigma\left(z^{[2]}\right)=\hat{y} \]

By stacking the parameters and outputs of different units into matrices and vectors, the network computes all hidden-unit activations without explicit loops. This vectorized structure is the foundation of efficient forward propagation in deeper neural networks.