Derivatives of Nonlinear Functions

A derivative describes the slope of a function at a particular point. For a straight line, the slope remains constant everywhere. For a nonlinear function, however, the slope can change depending on where it is measured.

This distinction is important in neural networks because most activation functions, loss functions, and cost functions are nonlinear. Their derivatives must therefore be evaluated at the current values of their inputs.

Reviewing a Constant Slope

Consider the linear function:\[ f(a)=3a \]

Its derivative is:\[ \frac{df(a)}{da}=3 \]

No matter where \(a\) is evaluated, increasing \(a\) by a small amount \(\Delta a\) changes the output by:\[ \Delta f=3\Delta a \]

For example, if:\[ \Delta a=0.001 \]

then:\[ \Delta f=0.003 \]

A straight line has the same slope everywhere.

A Nonlinear Example: \(f(a)=a^2\)

Now consider:\[ f(a)=a^2 \]

This is a curved function, so its slope changes from one point to another.

Let us first examine the function at:\[ a=2 \]

The function value is:\[ f(2)=2^2=4 \]

Now give \(a\) a small nudge:\[ a=2.001 \]

The new function value is:\[ f(2.001)=(2.001)^2 \]

Calculating it exactly:\[ (2.001)^2=4.004001 \]

This is approximately:\[ 4.004 \]

Estimating the Slope at \(a=2\)

The change in the input is:\[ \Delta a = 2.001-2 = 0.001 \]

The change in the output is:\[ \Delta f = 4.004001-4 = 0.004001 \]

The ratio is:\[ \frac{\Delta f}{\Delta a} = \frac{0.004001}{0.001} = 4.001 \]

For this small nudge, the slope is approximately:\[ 4 \]

Therefore, the derivative at \(a=2\) is:\[ \left. \frac{df(a)}{da} \right|_{a=2} = 4 \]

The notation means: evaluate the derivative of \(f(a)\) with respect to \(a\) at \(a=2\).

Why the Result Is Only Approximate

Using a finite change such as:\[ \Delta a=0.001 \]

produces an approximation of the derivative.

For \(f(a)=a^2\):\[ f(a+\Delta a) = (a+\Delta a)^2 \]

Expanding:\[ f(a+\Delta a) = a^2+2a\Delta a+(\Delta a)^2 \]

The change in the function is:\[ \Delta f = 2a\Delta a+(\Delta a)^2 \]

Dividing by \(\Delta a\):\[ \frac{\Delta f}{\Delta a} = 2a+\Delta a \]

At \(a=2\) and \(\Delta a=0.001\):\[ \frac{\Delta f}{\Delta a} = 4+0.001 = 4.001 \]

The extra \(0.001\) appears because the input change is small but not infinitesimal.

The formal derivative examines what happens as the change approaches zero:\[ \frac{df}{da} = \lim_{\Delta a\to0} \frac{f(a+\Delta a)-f(a)}{\Delta a} \]

As:\[ \Delta a\rightarrow0 \]

the extra \(\Delta a\) term disappears, leaving:\[ \frac{df}{da}=2a \]

The Derivative of \(a^2\)

The general derivative formula is:\[ \boxed{ \frac{d}{da}a^2=2a } \]

This means the slope depends on the current value of \(a\).

At:\[ a=2 \]

the slope is:\[ 2a=2\times2=4 \]

At:\[ a=5 \]

the slope is:\[ 2a=2\times5=10 \]

The curve is therefore steeper at \(a=5\) than at \(a=2\).

Checking the Slope at \(a=5\)

When:\[ a=5 \]

the function value is:\[ f(5)=5^2=25 \]

Now increase \(a\) to:\[ 5.001 \]

Then:\[ f(5.001) = (5.001)^2 = 25.010001 \]

The change in the function is:\[ \Delta f = 25.010001-25 = 0.010001 \]

The input changed by:\[ \Delta a=0.001 \]

Therefore:\[ \frac{\Delta f}{\Delta a} = \frac{0.010001}{0.001} = 10.001 \]

This is approximately:\[ 10 \]

The derivative formula confirms the exact local slope:\[ \left. \frac{df}{da} \right|_{a=5} = 2(5) = 10 \]

Slopes at Different Points

For the function:\[ f(a)=a^2 \]

the slope is:\[ f'(a)=2a \]

This gives different values at different points:\[ f'(0)=0 \]\[ f'(1)=2 \]\[ f'(2)=4 \]\[ f'(5)=10 \]

The graph becomes steeper as \(a\) becomes more positive.

The small triangle used to visualize the slope also changes shape. Its height-to-width ratio is 4 near \(a=2\), but 10 near \(a=5\).

The Tangent-Line Interpretation

For a nonlinear curve, the derivative is the slope of the tangent line at a particular point.

A tangent line touches the curve locally and points in the same direction as the curve at that point.

Near \(a=2\), the function \(a^2\) behaves approximately like a line with slope 4:\[ f(2+\Delta a) \approx f(2)+4\Delta a \]

Near \(a=5\), it behaves approximately like a line with slope 10:\[ f(5+\Delta a) \approx f(5)+10\Delta a \]

This local linear approximation is one of the most useful ways to interpret derivatives.

Predicting a Small Change with the Derivative

For a differentiable function:\[ \Delta f \approx f'(a)\Delta a \]

For \(f(a)=a^2\) at \(a=2\):\[ f'(2)=4 \]

If:\[ \Delta a=0.001 \]

then the derivative predicts:\[ \Delta f \approx 4(0.001) = 0.004 \]

The exact change is:\[ 0.004001 \]

The approximation is extremely close because the input change is small.

As \(\Delta a\) becomes smaller, the approximation becomes increasingly accurate.

The Power Rule

Calculus provides a general rule for powers of \(a\):\[ \boxed{ \frac{d}{da}a^n = na^{n-1} } \]

This is called the power rule.

For \(n=2\):\[ \frac{d}{da}a^2 = 2a \]

For \(n=3\):\[ \frac{d}{da}a^3 = 3a^2 \]

The derivative formula provides the slope at any selected point.

Another Example: \(f(a)=a^3\)

Consider:\[ f(a)=a^3 \]

Using the power rule:\[ \boxed{ \frac{df(a)}{da}=3a^2 } \]

At:\[ a=2 \]

the function value is:\[ f(2)=2^3=8 \]

The derivative is:\[ f'(2) = 3(2^2) = 3(4) = 12 \]

This means that near \(a=2\), a small change in \(a\) should produce approximately 12 times as much change in \(f(a)\).

Checking \(a^3\) Numerically

Increase the input from:\[ 2 \]

to:\[ 2.001 \]

Then:\[ f(2.001) = (2.001)^3 = 8.012006001 \]

The change in the output is:\[ \Delta f = 8.012006001-8 = 0.012006001 \]

The input changed by:\[ \Delta a=0.001 \]

Therefore:\[ \frac{\Delta f}{\Delta a} = 12.006001 \]

This is close to the derivative:\[ f'(2)=12 \]

The derivative-based approximation predicts:\[ \Delta f \approx 12(0.001) = 0.012 \]

which is very close to the exact change.

Why the Numerical Ratio Is Not Exactly 12

Expanding the cubic function explains the small difference:\[ (a+\Delta a)^3 = a^3 + 3a^2\Delta a + 3a(\Delta a)^2 + (\Delta a)^3 \]

Therefore:\[ \frac{f(a+\Delta a)-f(a)}{\Delta a} = 3a^2 + 3a\Delta a + (\Delta a)^2 \]

As:\[ \Delta a\rightarrow0 \]

the last two terms disappear:\[ 3a\Delta a\rightarrow0 \]\[ (\Delta a)^2\rightarrow0 \]

leaving:\[ f'(a)=3a^2 \]

The Natural Logarithm

Now consider:\[ f(a)=\log(a) \]

Here, \(\log\) means the natural logarithm, also written as:\[ \ln(a) \]

It is defined only for positive inputs:\[ a>0 \]

The derivative is:\[ \boxed{ \frac{d}{da}\log(a)=\frac{1}{a} } \]

This means the slope depends inversely on the current input.

Evaluating the Logarithm at \(a=2\)

When:\[ a=2 \]

the function value is approximately:\[ \log(2) \approx 0.693147 \]

Increase the input to:\[ a=2.001 \]

Then:\[ \log(2.001) \approx 0.693647 \]

The change is approximately:\[ \Delta f \approx 0.000500 \]

Because:\[ \Delta a=0.001 \]

the output changed by approximately half as much as the input.

Checking the Logarithm’s Derivative

The derivative at \(a=2\) is:\[ f'(2) = \frac{1}{2} = 0.5 \]

The derivative predicts:\[ \Delta f \approx f'(2)\Delta a \]\[ \Delta f \approx 0.5(0.001) = 0.0005 \]

This closely matches the actual change in the logarithm.

Geometrically, the tangent line at \(a=2\) has a slope of:\[ \frac{1}{2} \]

A horizontal change of \(0.001\) therefore produces a vertical change of approximately \(0.0005\).

How the Logarithm’s Slope Changes

For:\[ f(a)=\log(a) \]

the derivative is:\[ f'(a)=\frac{1}{a} \]

At different inputs:\[ f'(1)=1 \]\[ f'(2)=\frac{1}{2} \]\[ f'(5)=\frac{1}{5} \]\[ f'(10)=\frac{1}{10} \]

The logarithm continues increasing, but its slope becomes smaller as \(a\) grows. This matches the shape of the curve, which gradually becomes flatter.

Comparing the Examples

The examples demonstrate three different derivative patterns.

Linear function

\[ f(a)=3a \]\[ f'(a)=3 \]

The slope is constant everywhere.

Quadratic function

\[ f(a)=a^2 \]\[ f'(a)=2a \]

The slope changes linearly with \(a\).

Cubic function

\[ f(a)=a^3 \]\[ f'(a)=3a^2 \]

The slope depends on the square of \(a\).

Logarithmic function

\[ f(a)=\log(a) \]\[ f'(a)=\frac{1}{a} \]

The positive slope becomes smaller as \(a\) grows.

Derivative Reference Table

These formulas can be found in calculus references and derivative tables.

Numerical Approximation in Python

The small-nudge method can be used to approximate derivatives:

def approximate_derivative(f, a, epsilon=1e-6):
    return (
        f(a + epsilon) - f(a)
    ) / epsilon

For \(a^2\):

f = lambda a: a**2
print(approximate_derivative(f, 2.0))
# Approximately 4

For \(a^3\):

f = lambda a: a**3
print(approximate_derivative(f, 2.0))
# Approximately 12

For the natural logarithm:

import numpy as np
f = np.log
print(approximate_derivative(f, 2.0))
# Approximately 0.5

Using a smaller \(\epsilon\) generally improves the mathematical approximation until floating-point precision becomes a practical issue.

Why These Ideas Matter in Neural Networks

Neural networks contain many nonlinear functions. Their slopes depend on the values at which they are evaluated.

For example, the sigmoid derivative is:\[ \sigma'(z) = \sigma(z)(1-\sigma(z)) \]

The tanh derivative is:\[ \frac{d}{dz}\tanh(z) = 1-\tanh^2(z) \]

The ReLU derivative is:\[ \operatorname{ReLU}'(z) = \begin{cases} 0, & z<0 \\ 1, & z>0 \end{cases} \]

Backpropagation evaluates these derivatives at the current values of \(Z^{[l]}\). This determines how gradients flow backward through the network.

The fact that nonlinear functions have different slopes at different points is therefore central to neural-network learning.

Derivatives and Computation Graphs

A model may combine several functions:\[ a \rightarrow u=f(a) \rightarrow v=g(u) \rightarrow J=h(v) \]

To determine how \(a\) affects \(J\), backpropagation combines the derivatives of these intermediate operations.

A computation graph makes these relationships explicit and provides a systematic way to calculate the derivatives of complex functions.

This leads to the chain rule, one of the main mathematical ideas behind backpropagation.

Key Takeaway

A derivative is the slope of a function at a particular point. Unlike a straight line, a nonlinear function can have different slopes at different points.

For:\[ f(a)=a^2 \]

the derivative is:\[ f'(a)=2a \]

so:\[ f'(2)=4 \]

and:\[ f'(5)=10 \]

For:\[ f(a)=a^3 \]

the derivative is:\[ f'(a)=3a^2 \]

so:\[ f'(2)=12 \]

For:\[ f(a)=\log(a) \]

the derivative is:\[ f'(a)=\frac{1}{a} \]

so:\[ f'(2)=\frac{1}{2} \]

For a sufficiently small change:\[ \Delta f \approx f'(a)\Delta a \]

This provides an intuitive interpretation of a derivative: it predicts how much the output changes in response to a tiny change in the input. Neural-network backpropagation builds on exactly this idea to calculate how each parameter affects the final cost.