Derivatives

1. Purpose of Derivatives

In machine learning and optimization, derivatives are used to understand how a function changes with respect to its input.

A derivative represents:

the rate of change of a function
or more intuitively, the slope of the function at a specific point

---

2. Example Function

Consider the function:

$$f(a) = 3a$$

This is a linear function (a straight line).

---

3. Understanding Derivative Through Small Changes

Let’s analyze what happens when we slightly change the input.

Case 1: $a = 2$

• $f(2) = 6$

Now increase $a$ slightly:

• $a = 2.001$
• $f(2.001) = 6.003$

Observation:

• Change in input: $0.001$
• Change in output: $0.003$

$$\text{Slope} = \frac{0.003}{0.001} = 3$$

---

4. Interpretation

This means:

When $a$ increases by a small amount,
$f(a)$ increases 3 times as much

So the derivative is:

$$\frac{df(a)}{da} = 3$$

---

5. Another Point

Case 2: $a = 5$

• $f(5) = 15$

Increase slightly:

• $a = 5.001$
• $f(5.001) = 15.003$

Again:

$$\frac{0.003}{0.001} = 3$$

The slope is still 3

---

6. Key Property of Linear Functions

For the function $f(a) = 3a$:

• The derivative is constant
• The slope is the same at every point

$$\frac{df(a)}{da} = 3 \quad \text{for all } a$$

---

7. Meaning of Derivative

The derivative tells us:

If we slightly change the input, how much will the output change?

More precisely:

• A tiny change in input → proportional change in output
• The proportionality factor = derivative

---

8. Formal Definition

In theory, the derivative is defined using an infinitesimally small change in the input.

Instead of using a value like 0.001:

• We consider a change that is extremely close to zero

This gives a precise definition of the slope at a point.

---

9. Notation

Two common ways to write derivatives:

$$\frac{df(a)}{da}$$

or

$$\frac{d}{da} f(a)$$

Both represent the same concept:
the slope of $f(a)$ with respect to $a$

---

10. Importance in Machine Learning

Derivatives are essential because:

• They tell us the direction of change
• They are used in gradient descent
• They help determine how to update parameters