Sigmoid Function

The sigmoid maps any real number into \((0, 1)\):

\[ \sigma(x) = \frac{1}{1 + e^{-x}} \]

It is smooth, monotone, symmetric in the sense that \(\sigma(-x) = 1 - \sigma(x)\), and has the convenient derivative \(\sigma'(x) = \sigma(x)(1-\sigma(x))\). An input of 2 becomes \(\sigma(2) \approx 0.88\).

It is softmax with the second logit set to zero

import numpy as np

sigmoid = lambda x: 1 / (1 + np.exp(-x))
softmax = lambda z: np.exp(z - z.max()) / np.exp(z - z.max()).sum()

print("  the sigmoid is the two-class softmax, written with one number instead of two")
for x in (-2.0, 0.0, 0.7, 2.2, 4.0):
    print(f"    sigmoid({x:>5.1f}) = {sigmoid(x):.6f}   softmax([{x:>5.1f}, 0.0])[0] ="
          f" {softmax(np.array([x, 0.0]))[0]:.6f}")

print(f"\n  and it is symmetric: sigma(-x) = 1 - sigma(x)")
for x in (0.5, 1.5, 3.0):
    print(f"    sigma({-x:>4.1f}) = {sigmoid(-x):.6f}   1 - sigma({x:>3.1f}) = {1 - sigmoid(x):.6f}")

print("\n  exact and local input changes for an output decrease of 0.01")
for x in (0.0, 3.0, 6.0, 9.0):
    p = sigmoid(x)
    target = p - 0.01
    exact = np.log(target / (1 - target)) - x
    local = -0.01 / (p * (1 - p))
    print(f"    x={x:.1f}  exact delta={exact:.4f}  local estimate={local:.4f}"
          f"  upward 0.01 possible={p + 0.01 < 1}")
#   the sigmoid is the two-class softmax, written with one number instead of two
#     sigmoid( -2.0) = 0.119203   softmax([ -2.0, 0.0])[0] = 0.119203
#     sigmoid(  0.0) = 0.500000   softmax([  0.0, 0.0])[0] = 0.500000
#     sigmoid(  0.7) = 0.668188   softmax([  0.7, 0.0])[0] = 0.668188
#     sigmoid(  2.2) = 0.900250   softmax([  2.2, 0.0])[0] = 0.900250
#     sigmoid(  4.0) = 0.982014   softmax([  4.0, 0.0])[0] = 0.982014
#
#   and it is symmetric: sigma(-x) = 1 - sigma(x)
#     sigma(-0.5) = 0.377541   1 - sigma(0.5) = 0.377541
#     sigma(-1.5) = 0.182426   1 - sigma(1.5) = 0.182426
#     sigma(-3.0) = 0.047426   1 - sigma(3.0) = 0.047426
#
#   exact and local input changes for an output decrease of 0.01
#     x=0.0  exact delta=-0.0400  local estimate=-0.0400  upward 0.01 possible=True
#     x=3.0  exact delta=-0.2019  local estimate=-0.2214  upward 0.01 possible=True
#     x=6.0  exact delta=-1.6283  local estimate=-4.0543  upward 0.01 possible=False
#     x=9.0  exact delta=-4.4173  local estimate=-81.0508  upward 0.01 possible=False

The first block is an identity rather than an approximation: \(\sigma(x)\) equals \(\text{softmax}([x, 0])_0\) at every input, because \(\frac{e^x}{e^x + e^0} = \frac{1}{1 + e^{-x}}\).

That explains a convention that otherwise looks arbitrary. A binary classifier emits one number where a three-class one emits three, and the reason is that softmax is invariant to adding a constant to all its inputs — so with two classes you can always set the second logit to zero without losing anything. It is also why the binary logit is genuinely the log-odds while multi-class logits are not.

What saturation costs, in input units

The reciprocal derivative describes local sensitivity: for a sufficiently small output change, Δx is approximately Δp/σ′(x). A finite one-percentage-point move near saturation can fall outside that local approximation, and the direction matters because the probability cannot exceed 1.

For a decrease of exactly 0.01, compute Δx=logit(σ(x)−0.01)−x. At x=0, 6 and 9 this is about −0.0400, −1.6283 and −4.4173 respectively. An increase of 0.01 is impossible at x=6 or 9 because the starting probabilities already exceed 0.99. The code reports the exact decrease and the local linear estimate separately.

The sigmoid derivative at \(x = 9\) is small. The resulting parameter gradient also depends on the loss, inputs and other layers; saturation alone is not proof that the unit cannot learn. The activation contribution is discussed under squashing functions.

Near saturation, small input changes produce small output changes. That local fact helps explain difficult gradient propagation through repeated sigmoid activations, as discussed in squashing functions. It does not imply that the reciprocal derivative accurately predicts a large finite input move.

Where it is still the right choice

• The output of a binary classifier, and of each label in a multi-label model — there the outputs should not sum to one, so softmax would be wrong.
• Gates in recurrent architectures. LSTM and GRU gates need a number in \((0,1)\) meaning “how much of this to let through,” a smooth fraction that controls how much state passes through. Other bounded functions are possible, but they change the gate’s response and derivative.
• Logistic regression, where it is not an activation function at all but the inverse link that makes the model’s probabilities well defined.

One practical note in all three cases: when the loss is a fused function such as BCEWithLogitsLoss, pass it the raw logit and do not apply the sigmoid first. It computes the same quantity without the precision loss of going through a probability near 0 or 1. A loss that expects probabilities, such as BCELoss, takes the sigmoid output instead.

References

• PyTorch documentation, torch.nn.Sigmoid
• PyTorch documentation, BCEWithLogitsLoss