Squashing Function
Definition
A squashing function is a mathematical function that maps a wide (possibly infinite) range of input values into a limited, bounded range.
- It “squashes” large positive/negative inputs into a smaller interval.
- Common in neural networks as activation functions, especially when we want outputs like probabilities (between 0 and 1).
Properties of Squashing Functions
- Bounded range (e.g., [0,1] or [–1,1])
- Monotonic (output increases with input)
- Differentiable (needed for gradient descent)
- Helps prevent values from blowing up during training
Common Examples
- Sigmoid Function
$\sigma(x) = \frac{1}{1+e^{-x}}$
- Range: (0, 1)
- Used for probability outputs in logistic regression.
- Hyperbolic Tangent (tanh)
$\tanh(x) = \frac{e^x – e^{-x}}{e^x + e^{-x}}$
- Range: (–1, 1)
- Zero-centered, often preferred over sigmoid.
- Softmax Function
$\text{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j} e^{z_j}}$
- Squashes a vector into probabilities that sum to 1.
- Used in multi-class classification.
Why It’s Useful
- Probability interpretation: Squashing functions turn raw scores (logits) into probabilities.
- Stability in learning: Keeps outputs in a manageable range.
- Non-linearity: Allows neural networks to approximate complex functions.
Example
Suppose a model outputs a raw score $z = 5$.
- Sigmoid:
$\sigma(5) = \frac{1}{1+e^{-5}} \approx 0.993$
So, even though the score is large, it’s “squashed” into a probability close to 1.
In short:
A squashing function compresses arbitrary inputs into a bounded range (like 0–1), making it essential in probability modeling and neural networks.