Log-Space

1. Definition

• Saying we work in log-space means we represent values by their logarithms instead of the raw values.
• Instead of manipulating numbers $x$, we work with $\log(x)$.
• Useful when values span many orders of magnitude, or when computations risk numerical underflow/overflow.

---

2. Why Use Log-Space?

1. Avoiding underflow in probabilities
   • Probabilities like $10^{-100}$ can’t be represented precisely in floating-point.
   • Logs turn multiplication of small numbers into addition of logs:
     • $\log(p_1 \cdot p_2 \cdot p_3) = \log p_1 + \log p_2 + \log p_3$
   • Much safer numerically.
2. Simplifying products and exponentials
   • Bayesian models, HMMs, neural nets: log-likelihood is easier to compute.
   • Gradient optimization often uses log-likelihood because it’s smoother.
3. Interpreting data with heavy skew
   • Plotting values in log-space (e.g., log-scale on x or y axis) makes exponential trends linear.

---

3. Examples

Example 1: Multiplying probabilities

Say we want $p = 10^{-50} \times 10^{-60}$.

• In normal space: $10^{-110}$ → too small for double precision.
• In log-space:
  • $\log p = \log(10^{-50}) + \log(10^{-60}) = -50 \log 10 + -60 \log 10 = -110 \log 10$.
    → stays representable.

---

Example 2: Log-likelihood

In machine learning, we often maximize log-likelihood instead of likelihood:

$L(\theta) = \prod_{i=1}^n p(y_i|\theta), \quad \ell(\theta) = \log L(\theta) = \sum_{i=1}^n \log p(y_i|\theta)$

• Products become sums → easier to compute and optimize.

---

Example 3: Visualization

Population sizes by city: $[10, 10^3, 10^6, 10^8]$.

• Raw plot = skewed.
• In log-space (log of values): $[1,3,6,8]$ → nicely spread and linear-looking.

---

4. Common Phrases

• “Work in log-space” → store and manipulate $\log(x)$ instead of $x$.
• “Log-probabilities” → instead of storing $p$, store $\log(p)$.
• “Log-sum-exp trick” → numerical trick to add numbers in log-space safely:
  • $\log(e^a + e^b) = m + \log(e^{a-m} + e^{b-m}), \quad m=\max(a,b)$

---

Summary:
Log-space means working with the logarithm of values rather than the values themselves. It’s used to prevent numerical underflow/overflow, simplify multiplication into addition, and make skewed data easier to analyze. It’s fundamental in probability models, machine learning (log-likelihoods), and data visualization.