Underflow

Definition

Underflow happens when a computer tries to represent a very small positive number that is closer to zero than the smallest value it can store.

• The number gets rounded to zero because the machine’s floating-point precision cannot handle it.

In short: numbers too close to 0 “collapse” to 0 in computer arithmetic.

---

Why It Happens

Computers use finite precision floating-point numbers (IEEE 754 standard).

• Smallest positive normal number in 64-bit float ≈ $2.2 \times 10^{-308}$.
• If a calculation produces something smaller, e.g., $10^{-400}$, it cannot be represented → underflow → 0.

---

Examples

1. Multiplying Probabilities
   In machine learning, multiplying many probabilities (each < 1) quickly produces tiny numbers:
   • $P = 0.9^{1000} \approx 1.75 \times 10^{-46}$
   • If extended further, it could underflow to 0 in computer memory.
2. Exponential Functions
   • $\exp(-1000)$ is theoretically ≈ $3.7 \times 10^{-435}$.
     But in double precision, this will be stored as 0 (underflow).

---

Why It’s a Problem

• Can cause divide-by-zero errors or NaN results.
• In machine learning (e.g., softmax, likelihoods), underflow → probabilities become 0, breaking training.

---

How to Handle Underflow

1. Log Transformations (most common)
   • Instead of multiplying probabilities, sum their logs:
   • $\ln(p_1 p_2 \dots p_n) = \ln(p_1) + \ln(p_2) + \dots + \ln(p_n)$
   • This keeps numbers in a reasonable range.
2. Log-Sum-Exp Trick
   • Used in softmax computations to avoid both underflow and overflow.
   • $\text{softmax}(z_i) = \frac{e^{z_i – \max(z)}}{\sum_j e^{z_j – \max(z)}}$
   • Subtracting the maximum stabilizes values.
3. Use Higher Precision
   • Sometimes switch from 32-bit to 64-bit floating point.
4. Normalize Intermediately
   • In sequential multiplications, rescale numbers periodically.

---

Analogy

Imagine a scale that only measures down to 0.001 kg.

• If you put 0.0001 kg on it, the scale shows 0.
• That’s underflow: the value exists, but the system cannot represent it.

---

In short:
Underflow = when numbers are so tiny that the computer rounds them to zero.
It’s common in probability and machine learning, usually fixed with logarithms (log-space computation).