Neural Networks for Beginners: From Architecture to Forward Propagation

When I first studied neural networks, the hardest part was not any single equation. The real challenge was understanding how inputs, weights, biases, activation functions, and layers work together to produce a prediction.

This article follows an input through a small network, connecting the role of each parameter to the calculations that produce a prediction.

What is a neural network?

A neural network is a parameterized function that transforms an input into an output. The word parameterized means that part of the function is controlled by values—primarily weights and biases—that can be learned from data.

For a unit in the fully connected networks used here, the computation has two stages:

\[z=w^Tx+b\]

\[a=g(z)\]

• \(x\) is the input vector.
• \(w\) contains the weights applied to the inputs.
• \(b\) is the bias.
• \(z\) is the weighted sum plus bias, before activation.
• \(g\) is the activation function.
• \(a\) is the neuron’s output, also called its activation.

A network connects neurons in layers, each transforming the representation produced by the previous layer. With nonlinear activations between layers, it can represent relationships that a single affine transformation cannot. An affine transformation means a weighted sum plus a bias. Without an intervening nonlinear activation, two such layers combine into \(W_2(W_1x+b_1)+b_2=(W_2W_1)x+(W_2b_1+b_2)\), which is still one affine transformation. The third exercise checks this numerically.

A neural network as a composition of functions

Neural networks were loosely inspired by biological neurons, but treating them as mathematical functions is more useful for implementation. A network with two parameterized layers can be written as:

\[\hat{y}=f^{[2]}\left(f^{[1]}(x)\right)\]

The first function converts the original input into an intermediate representation. The second uses that representation to generate the final prediction. Deeper networks extend the same idea by composing more transformations.

The role of neural networks in supervised learning

In supervised learning, each training example contains an input \(x\) and a corresponding target \(y\):

\[(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),\ldots,(x^{(m)},y^{(m)})\]

Here, \(m\) is the number of examples. The network learns a function that maps each input to a useful prediction:

\[x\longrightarrow\text{neural network}\longrightarrow\hat{y}\]

During training, the model compares \(\hat{y}\) with \(y\) and adjusts its parameters to reduce the error. During inference, the trained parameters remain fixed while the model calculates predictions for new inputs.

Forward propagation calculates \(\hat{y}\). It does not improve the parameters by itself; the gradient-based training used in this series also needs a loss function, backward propagation, and a parameter update rule.

The basic structure of a neural network

In a feedforward network, information travels from inputs toward outputs without cycling back to an earlier computation. The network below has one hidden layer between its inputs and output. A direct input-to-output model is also feedforward; a hidden layer is not required by that term:

\[\text{input layer}\longrightarrow\text{hidden layer}\longrightarrow\text{output layer}\]

Input layer

The input layer represents the original features. If one example has two features, its input is \(x=[x_1,x_2]^T\). The input layer is not usually counted as a parameterized layer because it does not apply learned weights of its own.

Hidden layer

A hidden layer receives the previous layer’s values and creates a new representation. Each hidden unit has its own weights, allowing units to respond to different combinations of the same inputs. These values are internal activations computed from the input, rather than the input features or the final output. They are available to inspect, as the example below shows.

Output layer

The output layer converts the final hidden representation into the form required by the task. Binary classification may use one sigmoid output, multiclass classification may use one output per class, and regression may produce an unrestricted numerical value.

A sigmoid output lies between zero and one and can be used as a probability estimate for a binary class. This case is developed in Logistic Regression: A Complete Guide to Binary Classification.

Understanding neural-network notation

The notation in this series uses square brackets to identify layers and parentheses to identify examples. For layer \(l\), the main symbols are \(W^{[l]}\), \(b^{[l]}\), \(z^{[l]}\), and \(a^{[l]}\).

Defining \(a^{[0]}=x\) lets us write each fully connected layer in this network as:

\[z^{[l]}=W^{[l]}a^{[l-1]}+b^{[l]}\]

\[a^{[l]}=g^{[l]}\left(z^{[l]}\right)\]

Meanwhile, \(x^{(i)}\) and \(y^{(i)}\) refer to example \(i\). Therefore, \(a^{[1](3)}\) means the first layer’s activation for the third example.

Calculating the output of one neuron

Consider \(x=[2,3]^T\), \(w=[0.4,-0.2]^T\), and \(b=0.1\). The weighted sum is:

\[z=w^Tx+b=(0.4\times2)+(-0.2\times3)+0.1=0.3\]

The sigmoid function is \(\sigma(z)=1/(1+e^{-z})\). Applying it to this weighted sum gives \(a=\sigma(0.3)\approx0.5744\).

import numpy as np

x = np.array([[2.0], [3.0]])
w = np.array([[0.4], [-0.2]])
b = 0.1

z = w.T @ x + b
a = 1 / (1 + np.exp(-z))

print("z:", z.item())
print("a:", a.item())

z: 0.29999999999999993
a: 0.574442516811659

item() extracts the single number from a one-element array. The printed z is slightly below 0.3 because decimal fractions are represented approximately in floating-point arithmetic.

Forward propagation through a two-layer network

Consider a network with two inputs, three hidden units, and one output unit. The hidden layer uses tanh, which maps real inputs to values between −1 and 1, and the output layer uses sigmoid. Lowercase \(x,z,a\) describe one example; uppercase \(X,Z,A\) collect examples as columns. The input layer is not counted in the name “two-layer network”: the two parameterized layers are the hidden layer and the output layer. Here there is just one example, so each activation matrix has one column:

\[Z^{[1]}=W^{[1]}X+b^{[1]},\qquad A^{[1]}=\tanh\left(Z^{[1]}\right)\]

\[Z^{[2]}=W^{[2]}A^{[1]}+b^{[2]},\qquad A^{[2]}=\sigma\left(Z^{[2]}\right)=\hat{Y}\]

import numpy as np


def sigmoid(z):
    return 1 / (1 + np.exp(-z))


def forward_propagation(X, W1, b1, W2, b2):
    Z1 = W1 @ X + b1
    A1 = np.tanh(Z1)
    Z2 = W2 @ A1 + b2
    A2 = sigmoid(Z2)
    return Z1, A1, Z2, A2


X = np.array([[2.0], [3.0]])
W1 = np.array([[0.4, -0.2], [0.1, 0.5], [-0.3, 0.8]])
b1 = np.array([[0.1], [0.0], [-0.2]])
W2 = np.array([[0.7, -0.4, 0.2]])
b2 = np.array([[0.05]])

Z1, A1, Z2, prediction = forward_propagation(X, W1, b1, W2, b2)
print("Z1:", Z1.ravel())
print("A1:", np.round(A1.ravel(), 6))
print(f"Z2: {Z2.item():.7f}")
print(f"A2: {prediction.item():.7f}")

Z1: [0.3 1.7 1.6]
A1: [0.291313 0.935409 0.921669]
Z2: 0.0640889
A2: 0.5160167

The function returns the intermediate arrays along with the prediction so we can follow each stage. The first hidden unit has the same weighted sum, 0.3, as the single-neuron example. Applying tanh now gives about 0.2913, whereas sigmoid gave 0.5744. The other hidden weighted sums are \(0.1(2)+0.5(3)=1.7\) and \(-0.3(2)+0.8(3)-0.2=1.6\). The output combines their activations as \(0.7(0.291313)-0.4(0.935409)+0.2(0.921669)+0.05\approx0.064089\), then applies sigmoid to produce 0.5160167. The calls to ravel() provide one-dimensional views or copies for printing; the original Z1 and A1 arrays retain shape (3,1).

The output is produced by the chosen parameters, which have not been learned in this example. A loss compares it with a target; assessing predictive quality also requires evaluation on suitable data.

Checking the matrix dimensions

Let \(n^{[l]}\) be the number of units in layer \(l\), with \(n^{[0]}\) the input feature count, and let \(m\) be the batch size. Then \(W^{[l]}\in\mathbb{R}^{n^{[l]}\times n^{[l-1]}}\) and \(b^{[l]}\in\mathbb{R}^{n^{[l]}\times1}\). Weight rows index current-layer units and columns index preceding-layer units. Inputs have shape \(X\in\mathbb{R}^{n^{[0]}\times m}\), while \(Z^{[l]},A^{[l]}\in\mathbb{R}^{n^{[l]}\times m}\). NumPy broadcasts the bias column across the \(m\) example columns, adding the same bias to each example at a given unit.

A fully connected layer with \(n_{\text{in}}\) inputs and \(n_{\text{out}}\) units has \(n_{\text{out}}n_{\text{in}}\) weights and \(n_{\text{out}}\) biases. This network has \(3\times2+3=9\) hidden-layer parameters and \(1\times3+1=4\) output-layer parameters, for 13 total. These parameters are shared across examples; increasing the batch size adds activation values, not weights.

To run two examples together, place \((2,3)^T\) and \((-1,1)^T\) in adjacent columns. The same function then produces two prediction columns. Comparing with two separate calls checks that these fully connected layers and elementwise activations process each example independently:

X_batch = np.array([[2.0, -1.0], [3.0, 1.0]])
Z1_batch, A1_batch, Z2_batch, predictions = forward_propagation(
    X_batch, W1, b1, W2, b2
)
print("hidden shape:", A1_batch.shape)
print("output shape:", predictions.shape)
print("predictions:", np.round(predictions, 7))
separate = np.hstack([
    forward_propagation(X_batch[:, j:j+1], W1, b1, W2, b2)[-1]
    for j in range(X_batch.shape[1])
])
print("matches separate calls:", np.allclose(predictions, separate))
# hidden shape: (3, 2)
# output shape: (1, 2)
# predictions: [[0.5160167 0.429914 ]]
# matches separate calls: True

The slice j:j+1 keeps a one-column matrix, and np.hstack joins the separate output columns. The first prediction matches the earlier single-example result. Each hidden unit receives two input columns and adds its own bias to both; neither example changes the other’s prediction in this network.

What forward propagation does

Forward propagation applies the network’s current parameters to an input and produces a prediction. Each parameterized layer in this example applies an affine transformation followed by its activation function.

However, the forward pass does not learn by itself. Training continues by measuring the prediction error, propagating derivatives backward, and updating the parameters. Once I understood this repeated structure, the network became a sequence of small, traceable transformations rather than one large formula.

The computation-graphs article explains how to differentiate these operations during training.

A complete one-hidden-layer implementation of this is in Building a Shallow Neural Network with NumPy.

Exercises

1. Count the parameters. A network has layer sizes \([784,128,64,10]\). How many weights and biases in total? Now widen the first hidden layer to 256 and recompute. Which layer dominates, and why?

You should get: a number near 110,000, and one layer holding over 90% of it.

def count(dims):
    return sum(dims[i]*dims[i-1] + dims[i] for i in range(1, len(dims)))
print(count([784,128,64,10]))    # 109386
print(count([784,256,64,10]))    # 218058
print(784*128 + 128)             # 100480

2. Notation drill. For a network with \(n^{[0]}=5\), \(n^{[1]}=4\), \(n^{[2]}=1\) and a batch of \(m=32\), write the shape of \(W^{[1]},b^{[1]},Z^{[1]},A^{[1]},W^{[2]},Z^{[2]}\). Which of these change if the batch size changes?

You should get: six shapes, of which exactly half depend on \(m\).

3. Linear layers collapse. Build a two-layer network with no activation function: \(A^{[2]}=W^{[2]}(W^{[1]}X)\). Show numerically that a single matrix \(W\) reproduces its output to floating-point precision, and state what this implies about the functions such a stack can represent.

You should get: two outputs that match to floating-point precision, and one equivalent single layer.

import numpy as np
rng = np.random.default_rng(0)
W1, W2, X = rng.normal(size=(4,5)), rng.normal(size=(3,4)), rng.normal(size=(5,32))
two_layer = W2 @ (W1 @ X)
one_layer = (W2 @ W1) @ X
print(np.allclose(two_layer, one_layer), (W2 @ W1).shape)   # True (3, 5)