Understanding Computation Graphs

Neural-network training is organized into two complementary stages:

1. Forward propagation computes the network’s output and cost.
2. Backpropagation computes the derivatives needed to update its parameters.

A computation graph explains why these calculations naturally proceed in opposite directions. The forward pass moves from inputs to output, while the backward pass moves from the output back toward the inputs.

To understand this structure, consider a function that is much simpler than a complete neural network.

A Simple Function

Suppose we want to calculate:\[ J(a,b,c)=3(a+bc) \]

This function depends on three input variables:\[ a,\quad b,\quad c \]

Although it can be written as a single expression, the calculation can be divided into three simpler operations.

First, multiply \(b\) and \(c\):\[ u=bc \]

Next, add \(a\) and \(u\):\[ v=a+u \]

Finally, multiply \(v\) by 3:\[ J=3v \]

Combining these steps recovers the original function:\[ J=3v \]\[ =3(a+u) \]\[ =3(a+bc) \]

Why Break the Function into Steps?

The original expression is simple enough to calculate directly. However, a neural network may contain millions of individual operations.

Breaking a large calculation into small steps provides several advantages:

• Each operation is easy to understand.
• Intermediate values can be stored.
• The same operations can be reused.
• Dependencies between variables become explicit.
• Derivatives can be calculated systematically.
• Software can apply automatic differentiation.

A computation graph represents this sequence and its dependencies visually.

Constructing the Computation Graph

The first operation is:\[ u=bc \]

It receives \(b\) and \(c\) as inputs and produces \(u\):\[ (b,c) \longrightarrow u=bc \]

The second operation is:\[ v=a+u \]

It receives \(a\) and \(u\) and produces \(v\):\[ (a,u) \longrightarrow v=a+u \]

The final operation is:\[ J=3v \]

It receives \(v\) and produces the final result:\[ v \longrightarrow J=3v \]

The complete graph is:\[ \begin{aligned} b,c &\longrightarrow u=bc \\ a,u &\longrightarrow v=a+u \\ v &\longrightarrow J=3v \end{aligned} \]

Or as one sequence:\[ (a,b,c) \longrightarrow u=bc \longrightarrow v=a+u \longrightarrow J=3v \]

The variable \(a\) bypasses the first operation and enters at the addition step.

A Numerical Example

Let:\[ a=5,\qquad b=3,\qquad c=2 \]

Step 1: Calculate \(u\)

\[ u=bc \]\[ u=3\times2=6 \]

Step 2: Calculate \(v\)

\[ v=a+u \]\[ v=5+6=11 \]

Step 3: Calculate \(J\)

\[ J=3v \]\[ J=3\times11=33 \]

The same result is obtained from the original expression:\[ J=3(a+bc) \]\[ =3(5+3\times2) \]\[ =3(5+6) \]\[ =3(11)=33 \]

Forward Propagation

The left-to-right evaluation of the graph is called forward propagation.

Starting with:\[ a=5,\quad b=3,\quad c=2 \]

we calculate:\[ u=6 \]

then:\[ v=11 \]

and finally:\[ J=33 \]

The direction is:\[ (a,b,c) \rightarrow u \rightarrow v \rightarrow J \]

Each operation can be evaluated once all of its inputs are available.

Forward propagation therefore follows the dependency structure of the calculation.

Intermediate Values

The variables \(u\) and \(v\) are intermediate values:\[ u=bc \]\[ v=a+u \]

They are not the original inputs or final output, but they simplify the overall computation.

These values also become useful during backpropagation. Rather than treating \(J=3(a+bc)\) as one indivisible operation, we can differentiate each small operation separately and combine the results.

This is the central purpose of a computation graph.

The Distinguished Output

A computation graph is particularly useful when one output has a special role.

In this example, the distinguished output is:\[ J \]

In machine learning, \(J\) commonly represents the cost function:\[ J=J(W,b) \]

Training seeks parameter values that minimize this cost.

To update the parameters, we need derivatives such as:\[ \frac{\partial J}{\partial W} \]

and:\[ \frac{\partial J}{\partial b} \]

The computation graph shows how each parameter influences the final cost through a sequence of intermediate calculations.

Forward and Backward Directions

The two phases move in opposite directions.

Forward direction

Forward propagation calculates values:\[ \text{Inputs} \rightarrow \text{Intermediate values} \rightarrow \text{Output} \]

For the example:\[ (a,b,c) \rightarrow u \rightarrow v \rightarrow J \]

Backward direction

Backpropagation calculates sensitivities:\[ \text{Output} \rightarrow \text{Intermediate variables} \rightarrow \text{Inputs} \]

For the example:\[ J \rightarrow v \rightarrow u \rightarrow (a,b,c) \]

The backward pass determines how a small change in each variable would affect \(J\).

Local Derivatives of the Operations

Each node has simple local derivatives.

Multiplication node

For:\[ u=bc \]

the derivatives are:\[ \frac{\partial u}{\partial b}=c \]\[ \frac{\partial u}{\partial c}=b \]

Addition node

For:\[ v=a+u \]

the derivatives are:\[ \frac{\partial v}{\partial a}=1 \]\[ \frac{\partial v}{\partial u}=1 \]

Scaling node

For:\[ J=3v \]

the derivative is:\[ \frac{\partial J}{\partial v}=3 \]

Backpropagation combines these local derivatives using the chain rule.

Preview of the Backward Pass

Begin with the final operation:\[ J=3v \]

Its derivative is:\[ \frac{\partial J}{\partial v}=3 \]

Because:\[ v=a+u \]

we have:\[ \frac{\partial v}{\partial a}=1 \]

and:\[ \frac{\partial v}{\partial u}=1 \]

Using the chain rule:\[ \frac{\partial J}{\partial a} = \frac{\partial J}{\partial v} \frac{\partial v}{\partial a} \]\[ =3\times1=3 \]

Similarly:\[ \frac{\partial J}{\partial u} = \frac{\partial J}{\partial v} \frac{\partial v}{\partial u} \]\[ =3\times1=3 \]

Now use:\[ u=bc \]

For \(b\):\[ \frac{\partial J}{\partial b} = \frac{\partial J}{\partial u} \frac{\partial u}{\partial b} \]\[ =3c \]

For \(c\):\[ \frac{\partial J}{\partial c} = \frac{\partial J}{\partial u} \frac{\partial u}{\partial c} \]\[ =3b \]

Therefore:\[ \boxed{ \frac{\partial J}{\partial a}=3 } \]\[ \boxed{ \frac{\partial J}{\partial b}=3c } \]\[ \boxed{ \frac{\partial J}{\partial c}=3b } \]

At:\[ a=5,\quad b=3,\quad c=2 \]

these become:\[ \frac{\partial J}{\partial a}=3 \]\[ \frac{\partial J}{\partial b}=3(2)=6 \]\[ \frac{\partial J}{\partial c}=3(3)=9 \]

Interpreting the Derivatives

The derivative:\[ \frac{\partial J}{\partial a}=3 \]

means that a small increase in \(a\) produces approximately three times as much increase in \(J\).

The derivative:\[ \frac{\partial J}{\partial b}=6 \]

at \(c=2\) means that a small increase in \(b\) produces approximately six times as much increase in \(J\).

The derivative:\[ \frac{\partial J}{\partial c}=9 \]

at \(b=3\) means that a small increase in \(c\) produces approximately nine times as much increase in \(J\).

For a small change:\[ \Delta J \approx \frac{\partial J}{\partial a}\Delta a + \frac{\partial J}{\partial b}\Delta b + \frac{\partial J}{\partial c}\Delta c \]

This describes the local sensitivity of the output to all three inputs.

Verifying a Derivative Numerically

Consider the derivative with respect to \(b\).

At:\[ a=5,\quad b=3,\quad c=2 \]

the original value is:\[ J=33 \]

Increase \(b\) by:\[ \Delta b=0.001 \]

Then:\[ b=3.001 \]

The new output is:\[ J_{\text{new}} = 3(5+3.001\times2) \]\[ = 3(11.002) = 33.006 \]

The change is:\[ \Delta J=0.006 \]

Therefore:\[ \frac{\Delta J}{\Delta b} = \frac{0.006}{0.001} = 6 \]

This agrees with:\[ \frac{\partial J}{\partial b}=3c=6 \]

Why the Backward Pass Moves Right to Left

The final output depends directly on \(v\):\[ J=3v \]

The variable \(v\) depends on \(a\) and \(u\):\[ v=a+u \]

The variable \(u\) depends on \(b\) and \(c\):\[ u=bc \]

To calculate how \(b\) affects \(J\), we must follow the path:\[ b \rightarrow u \rightarrow v \rightarrow J \]

The chain rule combines the derivatives along this path. Computationally, the easiest approach is to begin with \(J\) and move backward:\[ J \rightarrow v \rightarrow u \rightarrow b \]

The same reasoning applies to \(c\).

This dependency structure is why backpropagation naturally moves from right to left.

Connection to Logistic Regression

Logistic regression can also be represented as a computation graph.

For one example:\[ z=w^Tx+b \]\[ a=\sigma(z) \]\[ \mathcal{L}(a,y) = -\left[ y\log(a)+(1-y)\log(1-a) \right] \]

The forward pass is:\[ (w,b,x) \rightarrow z \rightarrow a \rightarrow \mathcal{L} \]

The backward pass reverses the graph:\[ \mathcal{L} \rightarrow a \rightarrow z \rightarrow (w,b) \]

The result is the set of gradients needed for gradient descent.

Connection to Deep Neural Networks

A deep neural network contains a much larger computation graph:\[ X \rightarrow Z^{[1]} \rightarrow A^{[1]} \rightarrow Z^{[2]} \rightarrow A^{[2]} \rightarrow \cdots \rightarrow A^{[L]} \rightarrow J \]

Forward propagation evaluates the graph from left to right.

Backpropagation evaluates derivatives from right to left:\[ J \rightarrow dA^{[L]} \rightarrow dZ^{[L]} \rightarrow dA^{[L-1]} \rightarrow \cdots \]

At each stage, the network uses local derivatives and the chain rule.

Although a deep network’s graph is much larger, the underlying principle is the same as in:\[ J=3(a+bc) \]

Computation Graphs and Software

Modern deep-learning libraries construct computation graphs from operations such as:

• Matrix multiplication
• Addition
• Activation functions
• Summation
• Logarithms
• Loss calculations

During the forward pass, the system calculates and often stores intermediate values. During the backward pass, automatic differentiation applies the chain rule to calculate the gradients.

A simplified example in an automatic differentiation framework might look like:

u = b * c
v = a + u
J = 3 * v
J.backward()

The framework can then provide:

a.grad
b.grad
c.grad

representing:\[ \frac{\partial J}{\partial a}, \quad \frac{\partial J}{\partial b}, \quad \frac{\partial J}{\partial c} \]

Key Takeaway

A computation graph breaks a complex function into a sequence of simpler operations.

For:\[ J=3(a+bc) \]

the graph uses:\[ u=bc \]\[ v=a+u \]\[ J=3v \]

Forward propagation moves from the inputs to the result:\[ (a,b,c) \rightarrow u \rightarrow v \rightarrow J \]

Backpropagation moves in the opposite direction to calculate derivatives:\[ J \rightarrow v \rightarrow u \rightarrow (a,b,c) \]

This organization is not unique to the simple example. It is the same computational pattern used to train logistic regression and deep neural networks. Forward propagation calculates predictions and cost, while backpropagation uses the computation graph and chain rule to calculate the gradients required for learning.