Squashing Function

A squashing function maps an unbounded input into a bounded range. The sigmoid \(\sigma(x) = 1/(1+e^{-x})\) maps the real line onto \((0, 1)\); \(\tanh\) maps it onto \((-1, 1)\). Both are smooth and monotone with readily computed derivatives; gradient-based methods can also work with suitable nonsmooth functions.

A score of 5 becomes \(\sigma(5) \approx 0.993\) — large inputs are compressed toward the ceiling rather than passed through. That compression is the point, and it is also the problem.

(Softmax is often listed alongside these and belongs in a different category: it takes a whole vector and couples its components, where sigmoid and tanh apply to each number independently.)

Activation derivatives are only part of the gradient

import numpy as np

sigmoid = lambda x: 1 / (1 + np.exp(-x))
print(f"{'x':>6}  {'sigmoid(x)':>11} {'its slope':>10}   {'tanh(x)':>9} {'its slope':>10}   {'relu slope':>11}")
for x in (0.0, 1.0, 2.0, 5.0, 10.0):
    s = sigmoid(x)
    print(f"{x:>6.1f}  {s:>11.6f} {s * (1 - s):>10.6f}   {np.tanh(x):>9.6f}"
          f" {1 - np.tanh(x) ** 2:>10.6f}   {1.0:>11.1f}")

print("\n  activation-only derivative products, excluding weights and inactive paths")
print(f"{'layers':>8}  {'sigmoid activation max':>23}  {'tanh activation max':>20}  {'relu':>6}")
for depth in (1, 5, 10, 20, 40):
    print(f"{depth:>8}  {0.25 ** depth:>23.3e}  {1.0 ** depth:>20.3e}  {1.0 ** depth:>6.1f}")
print("\n  0.25 is the largest the sigmoid slope ever gets, at x = 0;"
      " away from zero it is far smaller")
#      x   sigmoid(x)  its slope     tanh(x)  its slope    relu slope
#    0.0     0.500000   0.250000    0.000000   1.000000           1.0
#    1.0     0.731059   0.196612    0.761594   0.419974           1.0
#    2.0     0.880797   0.104994    0.964028   0.070651           1.0
#    5.0     0.993307   0.006648    0.999909   0.000182           1.0
#   10.0     0.999955   0.000045    1.000000   0.000000           1.0
#
#   activation-only derivative products, excluding weights and inactive paths
#   layers   sigmoid activation max   tanh activation max    relu
#        1                2.500e-01             1.000e+00     1.0
#        5                9.766e-04             1.000e+00     1.0
#       10                9.537e-07             1.000e+00     1.0
#       20                9.095e-13             1.000e+00     1.0
#       40                8.272e-25             1.000e+00     1.0
#
#   0.25 is the largest the sigmoid slope ever gets, at x = 0; away from zero it is far smaller

Read the slope columns. The ReLU column uses its right-hand derivative at zero; the ordinary derivative is undefined at that point. The sigmoid’s derivative is \(\sigma(x)(1-\sigma(x))\), which peaks at 0.25 when \(x = 0\) and falls away in both directions — 0.0066 at \(x = 5\), 0.000045 at \(x = 10\).

The second table isolates the product of activation derivatives: ten factors of 0.25 give 9.537e−07 and forty give 8.272e−25. Actual backpropagation also multiplies weight Jacobians and depends on the active paths, so these are not universal bounds on a network’s complete gradient. The forty-factor value is representable and nonzero in float32; it is well above the smallest normal value, about 1.175e−38. Whether an update becomes negligible depends on its scale, the learning rate and the parameter being updated.

Repeated small activation derivatives contribute to vanishing gradients, but the weight matrices, initialization, normalization and architecture determine the full effect. In a scalar example, a weight of 4 followed by a sigmoid at zero has local derivative 4×0.25=1. This does not make deep sigmoid training generally easy; it shows why activation slopes alone are not the whole calculation.

tanh improves on it at the origin, where its slope is 1.0 rather than 0.25 — four times better, and the usual reason it was preferred. But it saturates faster once away from zero: at \(x = 5\) its slope is 0.000182 against the sigmoid’s 0.006648. Zero-centered outputs help; the saturation is the same disease.

For positive input ReLU contributes an activation derivative of 1, avoiding that source of attenuation along active paths. Weight Jacobians still multiply, and inactive units contribute zero. Ten active scalar layers with weights 0.5 have total derivative 0.5^10, not 1. ReLU is useful for gradient propagation, but it does not guarantee preserved gradients at every depth.

Where they still belong

• On the output layer. A binary classifier needs its score in \((0,1)\), and a multi-class one needs a distribution. The depth argument does not apply to a single final transformation.
• In gates. LSTM and GRU gates use sigmoids precisely because a value in \((0,1)\) is the natural way to express “let this fraction through.”
• In attention. In standard softmax attention, exponentiation and normalization produce positive weights that sum to one, making the output a weighted average. A raw ReLU does not provide that normalization; alternative weighting rules need their own normalization and behavior checks.
• Anywhere a bounded output is the requirement — a gate, a mixing weight, a probability — rather than a hidden representation.

So the rule that emerged is not “squashing functions are obsolete” but “use a squashing function where its role is needed, not as the default hidden activation of a deep feed-forward path.” Bounded outputs, gates and normalized mixtures need one wherever they sit in the network, while stacking sigmoids or tanh through many feed-forward layers invites the gradient attenuation described above. And where the output goes into a loss function, the squashing is usually best left to the loss itself, for the numerical reasons set out under logits.

References

• PyTorch documentation, torch.nn.Sigmoid
• PyTorch documentation, torch.nn.ReLU
• scikit-learn API reference, MLPClassifier
• NumPy documentation, finfo and floating-point limits