Derivatives
Calculus and derivatives appear frequently in machine learning because neural networks learn by measuring how small changes in their parameters affect the cost.
You do not need an advanced understanding of calculus to build and apply neural networks effectively. The most important starting intuition is simple:
A derivative describes the slope of a function—how much its output changes when its input changes by a very small amount.
Later, forward and backward functions will organize most of the required calculus into reusable operations. Still, understanding the basic meaning of a derivative makes gradient descent and backpropagation much easier to follow.
A Simple Linear Function
Consider the function:\[ f(a)=3a \]
This is a straight line. Its output is always three times its input.
For example, when:\[ a=2 \]
the output is:\[ f(2)=3\times2=6 \]
This gives the point:\[ (2,6) \]
on the graph.
Giving the Input a Small Nudge
Now increase \(a\) by a small amount:\[ a=2.001 \]
The change in the input is:\[ \Delta a = 2.001-2 = 0.001 \]
The new function value is:\[ f(2.001) = 3\times2.001 = 6.003 \]
The change in the output is:\[ \Delta f = 6.003-6 = 0.003 \]
Therefore:\[ \frac{\Delta f}{\Delta a} = \frac{0.003}{0.001} = 3 \]
When the input increases by \(0.001\), the output increases by \(0.003\), which is three times as much.
This ratio is the slope:\[ \text{Slope} = \frac{\text{change in output}} {\text{change in input}} \]
For this example:\[ \text{Slope}=3 \]
Derivative Means Slope
The word derivative can sound more complicated than the underlying idea. At an intuitive level, a derivative is simply the slope of a function at a particular point.
For:\[ f(a)=3a \]
the derivative is:\[ \frac{df(a)}{da}=3 \]
This means that if \(a\) changes by a very small amount, \(f(a)\) changes by approximately three times that amount.
If:\[ \Delta a=0.001 \]
then:\[ \Delta f\approx3\Delta a=0.003 \]
In this linear example, the relationship is exact rather than approximate.
Height Divided by Width
Geometrically, the slope can be visualized using a small triangle drawn along the line.
The triangle’s width is the change in the input:\[ \text{Width}=\Delta a \]
Its height is the change in the output:\[ \text{Height}=\Delta f \]
Therefore:\[ \text{Slope} = \frac{\text{Height}}{\text{Width}} = \frac{\Delta f}{\Delta a} \]
For the movement from \(a=2\) to \(a=2.001\):\[ \text{Slope} = \frac{6.003-6}{2.001-2} = \frac{0.003}{0.001} = 3 \]
The derivative tells us how steeply the line rises.
Checking the Slope at Another Point
Now consider:\[ a=5 \]
The function value is:\[ f(5)=3\times5=15 \]
Give \(a\) the same small nudge:\[ a=5.001 \]
Then:\[ f(5.001) = 3\times5.001 = 15.003 \]
The input changed by:\[ \Delta a = 5.001-5 = 0.001 \]
The output changed by:\[ \Delta f = 15.003-15 = 0.003 \]
The slope is again:\[ \frac{\Delta f}{\Delta a} = \frac{0.003}{0.001} = 3 \]
Thus, the derivative at \(a=5\) is also:\[ \frac{df(a)}{da}=3 \]
Why the Slope Is the Same Everywhere
The graph of:\[ f(a)=3a \]
is a straight line. Its steepness never changes.
No matter where we choose two nearby points:\[ \frac{\Delta f}{\Delta a}=3 \]
Therefore:\[ f'(a)=3 \]
for every value of \(a\).
The notation \(f'(a)\), pronounced “\(f\) prime of \(a\),” is another way to write the derivative:\[ f'(a) = \frac{df(a)}{da} = 3 \]
This means the function has a constant slope of 3 everywhere.
Common Derivative Notation
The derivative of \(f(a)\) with respect to \(a\) can be written in several equivalent ways:\[ \frac{df(a)}{da} \]\[ \frac{d}{da}f(a) \]\[ f'(a) \]
In this example:\[ \frac{df(a)}{da} = \frac{d}{da}f(a) = f'(a) = 3 \]
The expression “with respect to \(a\)” means that \(a\) is the input being changed.
From a Small Change to an Infinitesimal Change
The example used a change of:\[ \Delta a=0.001 \]
This is useful for intuition, but the formal derivative uses a change that approaches zero.
The derivative is defined as:\[ f'(a) = \lim_{\Delta a\to0} \frac{f(a+\Delta a)-f(a)}{\Delta a} \]
The expression:\[ f(a+\Delta a)-f(a) \]
measures the change in the output, while \(\Delta a\) measures the change in the input.
The limit asks what this ratio approaches as the input change becomes arbitrarily small.
For:\[ f(a)=3a \]
we can substitute directly:\[ f(a+\Delta a) = 3(a+\Delta a) = 3a+3\Delta a \]
Therefore:\[ \frac{f(a+\Delta a)-f(a)}{\Delta a} = \frac{3a+3\Delta a-3a}{\Delta a} \]\[ = \frac{3\Delta a}{\Delta a} = 3 \]
Since the ratio is already 3 for every nonzero \(\Delta a\), taking the limit gives:\[ f'(a)=3 \]
Local Sensitivity
Another useful interpretation of a derivative is sensitivity.
The derivative tells us how sensitive the output is to a small change in the input.
For:\[ f(a)=3a \]
the sensitivity is 3:\[ \frac{df}{da}=3 \]
This means:
- Increasing \(a\) by \(0.001\) increases \(f\) by approximately \(0.003\).
- Increasing \(a\) by \(0.01\) increases \(f\) by approximately \(0.03\).
- Decreasing \(a\) by \(0.001\) decreases \(f\) by approximately \(0.003\).
For a small input change:\[ \Delta f \approx \frac{df}{da}\Delta a \]
In this example:\[ \Delta f\approx3\Delta a \]
Again, because the function is linear, this relationship is exact for changes of any size.
Positive, Negative, and Zero Derivatives
Although this example has a derivative of 3, derivatives can have other signs.
Positive derivative
If:\[ f'(a)>0 \]
then the function increases as \(a\) increases.
Negative derivative
If:\[ f'(a)<0 \]
then the function decreases as \(a\) increases.
Zero derivative
If:\[ f'(a)=0 \]
then the function is locally flat at that point.
For:\[ f(a)=3a \]
the derivative is positive everywhere:\[ f'(a)=3>0 \]
so the function always increases from left to right.
Why Derivatives Matter in Machine Learning
A machine-learning cost function depends on model parameters.
For example:\[ J=J(w,b) \]
To reduce the cost, we need to know how it changes when \(w\) or \(b\) changes slightly.
The relevant derivatives are:\[ \frac{\partial J}{\partial w} \]
and:\[ \frac{\partial J}{\partial b} \]
They measure the sensitivity of the cost to the parameters.
Gradient descent uses them to update the model:\[ w:=w-\alpha\frac{\partial J}{\partial w} \]\[ b:=b-\alpha\frac{\partial J}{\partial b} \]
If a derivative is positive, subtracting it decreases the parameter. If it is negative, subtracting it increases the parameter. In both cases, the goal is to move toward a lower cost.
Derivatives in Neural Networks
A neural network applies many functions in sequence:\[ X \rightarrow Z^{[1]} \rightarrow A^{[1]} \rightarrow Z^{[2]} \rightarrow A^{[2]} \rightarrow J \]
Backpropagation uses derivatives to determine how a small change in each intermediate value would affect the final cost.
For example:\[ dW^{[l]} = \frac{\partial J}{\partial W^{[l]}} \]
describes how the cost responds to a small change in the weights of layer \(l\).
Although the complete calculation can involve many derivatives, neural-network implementations organize them into reusable forward and backward functions. Once these components are implemented, the underlying calculus does not need to be derived again during every use of the model.
Forward and Backward Functions
A forward function calculates a layer’s output:\[ Z^{[l]} = W^{[l]}A^{[l-1]}+b^{[l]} \]\[ A^{[l]} = g^{[l]}(Z^{[l]}) \]
The corresponding backward function calculates the necessary derivatives:\[ dZ^{[l]} \]\[ dW^{[l]} \]\[ db^{[l]} \]\[ dA^{[l-1]} \]
These functions encapsulate much of the calculus required for neural-network training.
As a result, understanding the intuition behind slope and sensitivity is often sufficient to use the algorithms effectively, even without an advanced background in calculus.
Numerical Approximation of a Derivative
The small-nudge idea can also be implemented directly.
For:\[ f(a)=3a \]
choose a small value:\[ \epsilon=0.001 \]
Then approximate the derivative:\[ f'(a) \approx \frac{f(a+\epsilon)-f(a)}{\epsilon} \]
In Python:
def f(a): return 3 * aa = 2.0epsilon = 0.001approximate_derivative = ( f(a + epsilon) - f(a)) / epsilonprint(approximate_derivative)
The result is approximately:
3.0
This finite-difference method is closely related to gradient checking, a technique used to verify backpropagation implementations.
Linear and Nonlinear Functions
A straight line has the same derivative everywhere. For:\[ f(a)=3a \]
the slope is always:\[ 3 \]
Nonlinear functions behave differently. Their slopes can change from one point to another.
For example, if:\[ f(a)=a^2 \]
the function is relatively flat near \(a=0\) and much steeper for larger values of \(a\). Its derivative therefore depends on where it is evaluated.
This point-dependent slope is essential in neural networks because activation functions and cost functions are generally nonlinear.
Key Takeaway
A derivative measures how much a function’s output changes when its input changes by an extremely small amount.
For:\[ f(a)=3a \]
increasing \(a\) by:\[ 0.001 \]
increases \(f(a)\) by:\[ 0.003 \]
Therefore, the slope is:\[ \frac{0.003}{0.001}=3 \]
and:\[ \boxed{ \frac{df(a)}{da}=3 } \]
Because this function is a straight line, its slope is 3 everywhere.
In machine learning, derivatives measure how sensitive the cost is to model parameters. Gradient descent and backpropagation use this information to decide how those parameters should change. A basic intuition for derivatives as slopes is therefore enough to begin understanding how neural networks learn.