Computing Derivatives with a Computation Graph

A computation graph organizes a function into a sequence of simple operations. It supports two complementary calculations:

• A left-to-right forward pass computes the value of the function.
• A right-to-left backward pass computes its derivatives.

The backward calculation is efficient because derivatives computed near the output can be reused to calculate derivatives of earlier variables.

Reviewing the Computation Graph

Consider the function:\[ J=3(a+bc) \]

Instead of evaluating it as one expression, break it into three steps:\[ u=bc \]\[ v=a+u \]\[ J=3v \]

The resulting computation graph is:\[ (b,c)\rightarrow u \]\[ (a,u)\rightarrow v \]\[ v\rightarrow J \]

Equivalently:\[ (a,b,c) \rightarrow u=bc \rightarrow v=a+u \rightarrow J=3v \]

The variable \(a\) enters at the second operation, where it is added to \(u\).

Forward Propagation

Suppose:\[ a=5,\qquad b=3,\qquad c=2 \]

The forward pass moves from left to right.

First:\[ u=bc=3\times2=6 \]

Next:\[ v=a+u=5+6=11 \]

Finally:\[ J=3v=3\times11=33 \]

Thus:\[ \boxed{J=33} \]

Forward propagation calculates the output by following the dependencies in the graph:\[ (a,b,c)\rightarrow u\rightarrow v\rightarrow J \]

Beginning Backpropagation

Backpropagation moves through the graph in the opposite direction:\[ J\rightarrow v\rightarrow(a,u)\rightarrow(b,c) \]

Its goal is to calculate how sensitive \(J\) is to every intermediate variable and input:\[ \frac{\partial J}{\partial v}, \quad \frac{\partial J}{\partial a}, \quad \frac{\partial J}{\partial u}, \quad \frac{\partial J}{\partial b}, \quad \frac{\partial J}{\partial c} \]

Each derivative answers a question of the following form:

If this variable changes by a tiny amount, approximately how much will \(J\) change?

Computing \(\frac{\partial J}{\partial v}\)

The final operation is:\[ J=3v \]

At the current point:\[ v=11 \]

and:\[ J=33 \]

Suppose we increase \(v\) slightly:\[ v=11.001 \]

The new value of \(J\) is:\[ J=3(11.001)=33.003 \]

The changes are:\[ \Delta v=0.001 \]

and:\[ \Delta J=0.003 \]

Therefore:\[ \frac{\Delta J}{\Delta v} = \frac{0.003}{0.001} = 3 \]

The derivative is:\[ \boxed{ \frac{\partial J}{\partial v}=3 } \]

This is identical to the familiar derivative:\[ \frac{d}{dv}(3v)=3 \]

We have now completed the first step of backpropagation.

Derivative Notation in Code

When implementing backpropagation, the final output is usually a variable such as the cost \(J\). Most derivative variables therefore represent the derivative of \(J\) with respect to another quantity.

Instead of using a long variable name such as:

dJ_dv

a common convention is:

dv

In this notation:\[ dv \equiv \frac{\partial J}{\partial v} \]

Therefore:\[ dv=3 \]

The same convention applies to the other variables:\[ da \equiv \frac{\partial J}{\partial a} \]\[ du \equiv \frac{\partial J}{\partial u} \]\[ db \equiv \frac{\partial J}{\partial b} \]\[ dc \equiv \frac{\partial J}{\partial c} \]

The prefix d means “the derivative of the final output with respect to this variable.”

Computing \(\frac{\partial J}{\partial a}\)

We now want to know how changing \(a\) affects \(J\).

At the current point:\[ a=5 \]

Increase it slightly:\[ a=5.001 \]

Because:\[ v=a+u \]

and \(u=6\), the new value of \(v\) is:\[ v=5.001+6=11.001 \]

Then:\[ J=3(11.001)=33.003 \]

Therefore:\[ \Delta a=0.001 \]

produces:\[ \Delta J=0.003 \]

The derivative is:\[ \boxed{ \frac{\partial J}{\partial a}=3 } \]

In code:

da = 3

Understanding the Chain of Effects

Changing \(a\) does not affect \(J\) directly. It first changes \(v\), which then changes \(J\):\[ a\rightarrow v\rightarrow J \]

The effect can be separated into two local relationships:\[ \frac{\partial v}{\partial a} \]

and:\[ \frac{\partial J}{\partial v} \]

Because:\[ v=a+u \]

we have:\[ \frac{\partial v}{\partial a}=1 \]

We already calculated:\[ \frac{\partial J}{\partial v}=3 \]

The total effect is found by multiplying these derivatives:\[ \frac{\partial J}{\partial a} = \frac{\partial J}{\partial v} \frac{\partial v}{\partial a} \]

Therefore:\[ \frac{\partial J}{\partial a} = 3\times1 = 3 \]

This multiplication of derivatives is the chain rule.

The Chain Rule

If one variable affects a second variable, which then affects a third:\[ a\rightarrow v\rightarrow J \]

then:\[ \boxed{ \frac{\partial J}{\partial a} = \frac{\partial J}{\partial v} \frac{\partial v}{\partial a} } \]

The chain rule combines the local effects along a path through the computation graph.

In words:

• How much does \(a\) change \(v\)?
• How much does \(v\) change \(J\)?
• Multiply those effects to determine how much \(a\) changes \(J\).

This is the central mathematical principle behind backpropagation.

Computing \(\frac{\partial J}{\partial u}\)

Next, consider the variable \(u\).

The relevant path is:\[ u\rightarrow v\rightarrow J \]

Because:\[ v=a+u \]

the local derivative is:\[ \frac{\partial v}{\partial u}=1 \]

We already know:\[ \frac{\partial J}{\partial v}=3 \]

Applying the chain rule:\[ \frac{\partial J}{\partial u} = \frac{\partial J}{\partial v} \frac{\partial v}{\partial u} \]\[ = 3\times1 = 3 \]

Therefore:\[ \boxed{ \frac{\partial J}{\partial u}=3 } \]

In code:

du = 3

Verifying \(\frac{\partial J}{\partial u}\) Numerically

The original value is:\[ u=6 \]

Increase it to:\[ u=6.001 \]

Then:\[ v=a+u=5+6.001=11.001 \]

and:\[ J=3(11.001)=33.003 \]

Thus:\[ \Delta u=0.001 \]

produces:\[ \Delta J=0.003 \]

Therefore:\[ \frac{\Delta J}{\Delta u}=3 \]

which confirms:\[ \frac{\partial J}{\partial u}=3 \]

Reusing Previously Computed Derivatives

Notice that both \(a\) and \(u\) influence \(J\) through \(v\):\[ a\rightarrow v\rightarrow J \]\[ u\rightarrow v\rightarrow J \]

Once we have computed:\[ \frac{\partial J}{\partial v}=3 \]

we can reuse it:\[ \frac{\partial J}{\partial a} = \frac{\partial J}{\partial v} \frac{\partial v}{\partial a} \]

and:\[ \frac{\partial J}{\partial u} = \frac{\partial J}{\partial v} \frac{\partial v}{\partial u} \]

This reuse is one reason why right-to-left backpropagation is efficient.

Computing \(\frac{\partial J}{\partial b}\)

The variable \(b\) affects \(J\) through the following path:\[ b\rightarrow u\rightarrow v\rightarrow J \]

Because:\[ u=bc \]

the local derivative of \(u\) with respect to \(b\) is:\[ \frac{\partial u}{\partial b}=c \]

At the current value:\[ c=2 \]

so:\[ \frac{\partial u}{\partial b}=2 \]

We already know:\[ \frac{\partial J}{\partial u}=3 \]

Applying the chain rule:\[ \frac{\partial J}{\partial b} = \frac{\partial J}{\partial u} \frac{\partial u}{\partial b} \]

Therefore:\[ \frac{\partial J}{\partial b} = 3\times2 = 6 \]

Thus:\[ \boxed{ \frac{\partial J}{\partial b}=6 } \]

In code:

db = 6

Verifying \(\frac{\partial J}{\partial b}\) Numerically

Originally:\[ b=3 \]

Increase \(b\) by:\[ 0.001 \]

so that:\[ b=3.001 \]

Because:\[ u=bc \]

and \(c=2\):\[ u=3.001\times2=6.002 \]

The change in \(u\) is:\[ \Delta u=0.002 \]

This is twice the change in \(b\), confirming:\[ \frac{\partial u}{\partial b}=2 \]

Next:\[ v=a+u=5+6.002=11.002 \]

Finally:\[ J=3v=3(11.002)=33.006 \]

The change in \(J\) is:\[ \Delta J=0.006 \]

Therefore:\[ \frac{\Delta J}{\Delta b} = \frac{0.006}{0.001} = 6 \]

This agrees with the derivative:\[ \frac{\partial J}{\partial b}=6 \]

Breaking Down the Effect of \(b\)

The total effect consists of two stages.

First, changing \(b\) changes \(u\):\[ \Delta u \approx \frac{\partial u}{\partial b}\Delta b \]

At \(c=2\):\[ \Delta u \approx 2\Delta b \]

Next, changing \(u\) changes \(J\):\[ \Delta J \approx \frac{\partial J}{\partial u}\Delta u \]

Since:\[ \frac{\partial J}{\partial u}=3 \]

we get:\[ \Delta J \approx 3(2\Delta b) = 6\Delta b \]

Thus:\[ \frac{\partial J}{\partial b}=6 \]

The chain rule multiplies the local sensitivities along the path.

Computing \(\frac{\partial J}{\partial c}\)

The variable \(c\) also affects \(J\) through \(u\):\[ c\rightarrow u\rightarrow v\rightarrow J \]

Because:\[ u=bc \]

the local derivative is:\[ \frac{\partial u}{\partial c}=b \]

At:\[ b=3 \]

we have:\[ \frac{\partial u}{\partial c}=3 \]

We already calculated:\[ \frac{\partial J}{\partial u}=3 \]

Therefore:\[ \frac{\partial J}{\partial c} = \frac{\partial J}{\partial u} \frac{\partial u}{\partial c} \]\[ = 3\times3 = 9 \]

Thus:\[ \boxed{ \frac{\partial J}{\partial c}=9 } \]

In code:

dc = 9

Verifying \(\frac{\partial J}{\partial c}\)

Increase \(c\) from:\[ 2 \]

to:\[ 2.001 \]

Then:\[ u=bc=3(2.001)=6.003 \]

Next:\[ v=a+u=5+6.003=11.003 \]

Finally:\[ J=3v=3(11.003)=33.009 \]

Therefore:\[ \Delta c=0.001 \]

produces:\[ \Delta J=0.009 \]

The ratio is:\[ \frac{\Delta J}{\Delta c} = \frac{0.009}{0.001} = 9 \]

This confirms:\[ \frac{\partial J}{\partial c}=9 \]

Complete Set of Derivatives

For:\[ J=3(a+bc) \]

the derivatives are:\[ \frac{\partial J}{\partial v}=3 \]\[ \frac{\partial J}{\partial a}=3 \]\[ \frac{\partial J}{\partial u}=3 \]\[ \frac{\partial J}{\partial b}=3c \]\[ \frac{\partial J}{\partial c}=3b \]

At:\[ a=5,\qquad b=3,\qquad c=2 \]

the numerical values are:\[ dv=3 \]\[ da=3 \]\[ du=3 \]\[ db=6 \]\[ dc=9 \]

Local Derivative Rules

Each node in the graph performs a simple operation with simple derivatives.

Scaling

For:\[ J=3v \]

the local derivative is:\[ \frac{\partial J}{\partial v}=3 \]

Addition

For:\[ v=a+u \]

the local derivatives are:\[ \frac{\partial v}{\partial a}=1 \]\[ \frac{\partial v}{\partial u}=1 \]

An addition node passes the incoming derivative backward unchanged to both inputs.

Multiplication

For:\[ u=bc \]

the local derivatives are:\[ \frac{\partial u}{\partial b}=c \]\[ \frac{\partial u}{\partial c}=b \]

A multiplication node passes the incoming derivative to each input after multiplying it by the other input.

Backpropagation as Repeated Local Calculations

The backward pass can be implemented using the derivative arriving from the right.

Start with:\[ dv=3 \]

Through the addition node:\[ da=dv\times1=3 \]\[ du=dv\times1=3 \]

Through the multiplication node:\[ db=du\times c \]\[ dc=du\times b \]

Substituting \(b=3\), \(c=2\), and \(du=3\):\[ db=3\times2=6 \]\[ dc=3\times3=9 \]

This provides a compact algorithm:

# Forward pass
u = b * c
v = a + u
J = 3 * v
# Backward pass
dv = 3
da = dv
du = dv
db = du * c
dc = du * b

Why Backpropagation Is Efficient

Suppose each derivative were calculated independently from the original expression:\[ J=3(a+bc) \]

For a small graph, that would be manageable. In a large neural network, however, repeatedly retracing the same paths would waste substantial computation.

Backpropagation avoids this duplication.

Once it computes:\[ du=\frac{\partial J}{\partial u} \]

that value is reused to calculate both:\[ db=\frac{\partial J}{\partial b} \]

and:\[ dc=\frac{\partial J}{\partial c} \]

Similarly, the previously computed value:\[ dv=\frac{\partial J}{\partial v} \]

is reused to calculate:\[ da \]

and:\[ du \]

The efficient order is therefore:\[ dv \rightarrow (da,du) \rightarrow (db,dc) \]

This is reverse-mode automatic differentiation, the method used by backpropagation.

Forward and Backward Passes Compared

Forward pass

The forward pass computes values:\[ u=bc \]\[ v=a+u \]\[ J=3v \]

Its direction is:\[ (a,b,c)\rightarrow u\rightarrow v\rightarrow J \]

Backward pass

The backward pass computes sensitivities:\[ dv=3 \]\[ da=3,\qquad du=3 \]\[ db=6,\qquad dc=9 \]

Its direction is:\[ J\rightarrow v\rightarrow u\rightarrow(b,c) \]

with a separate branch from \(v\) to \(a\).

Connection to Gradient Descent

In a machine-learning problem, \(J\) usually represents the cost that should be minimized.

Suppose \(a\), \(b\), and \(c\) were trainable parameters. Gradient descent would update them as:\[ a:=a-\alpha da \]\[ b:=b-\alpha db \]\[ c:=c-\alpha dc \]

where \(\alpha\) is the learning rate.

At the current values:\[ da=3,\qquad db=6,\qquad dc=9 \]

the updates would be:\[ a:=a-3\alpha \]\[ b:=b-6\alpha \]\[ c:=c-9\alpha \]

The derivatives tell gradient descent how the cost changes in response to each parameter.

Connection to Logistic Regression

Logistic regression has a more complicated graph:\[ (w,b,x) \rightarrow z=w^Tx+b \rightarrow a=\sigma(z) \rightarrow \mathcal{L}(a,y) \]

Forward propagation calculates the prediction and loss.

Backpropagation moves from the loss toward the parameters:\[ \mathcal{L} \rightarrow a \rightarrow z \rightarrow (w,b) \]

As in the simple example, it computes local derivatives and combines them using the chain rule.

Connection to Neural Networks

A deep neural network contains many repeated operations:\[ X \rightarrow Z^{[1]} \rightarrow A^{[1]} \rightarrow Z^{[2]} \rightarrow A^{[2]} \rightarrow \cdots \rightarrow J \]

Forward propagation evaluates these operations from left to right.

Backpropagation calculates:\[ dA^{[L]}, \quad dZ^{[L]}, \quad dW^{[L]}, \quad db^{[L]} \]

and then proceeds through the preceding layers:\[ dA^{[L-1]}, \quad dZ^{[L-1]}, \quad dW^{[L-1]}, \quad db^{[L-1]} \]

The graph may be much larger, but the logic remains the same:

1. Calculate each operation during the forward pass.
2. Store useful intermediate values.
3. Begin with the final output during the backward pass.
4. Multiply by local derivatives while moving backward.
5. Reuse previously computed derivatives.

Key Takeaway

For the function:\[ J=3(a+bc) \]

forward propagation calculates:\[ u=bc \]\[ v=a+u \]\[ J=3v \]

At:\[ a=5,\quad b=3,\quad c=2 \]

the result is:\[ J=33 \]

Backpropagation moves from right to left and calculates:\[ \boxed{dv=3} \]\[ \boxed{da=3} \]\[ \boxed{du=3} \]\[ \boxed{db=6} \]\[ \boxed{dc=9} \]

The chain rule connects the derivatives along each path:\[ \frac{\partial J}{\partial a} = \frac{\partial J}{\partial v} \frac{\partial v}{\partial a} \]\[ \frac{\partial J}{\partial b} = \frac{\partial J}{\partial u} \frac{\partial u}{\partial b} \]\[ \frac{\partial J}{\partial c} = \frac{\partial J}{\partial u} \frac{\partial u}{\partial c} \]

The most efficient way to compute all these derivatives is to move backward through the graph, reusing previously calculated values. This is the central computational idea behind backpropagation in logistic regression and deep neural networks.