Implementing Dropout Regularization

ByJu Yeon Eum

Dropout is a powerful regularization technique for reducing overfitting in neural networks. During training, it randomly disables selected units and their connections, causing the model to work with a different reduced network on each iteration.

The most common implementation is called inverted dropout. It scales the retained activations during training so that evaluation can use the complete network without additional adjustment.

How Dropout Changes a Neural Network

Suppose a neural network is overfitting. Dropout assigns each hidden unit a probability of being retained.

For example, with:\[ \text{keep\_prob}=0.5 \]

each unit has:

  • A 50% probability of being retained
  • A 50% probability of being removed

After randomly selecting the units, all connections to and from the removed units are effectively disabled for that pass.

The resulting network is smaller than the original network.

Dropout trains a randomly reduced version of the neural network on each pass.

The selection is repeated, so later iterations normally use different subnetworks.

A New Subnetwork on Each Iteration

Dropout does not permanently remove units from the architecture. Instead, it generates a temporary binary mask during training.

For one iteration, the active units might be:\[ \begin{bmatrix} 1 & 0 & 1 & 1 \end{bmatrix} \]

On another iteration, the mask might be:\[ \begin{bmatrix} 0 & 1 & 1 & 0 \end{bmatrix} \]

A value of 1 means that the unit is retained, while 0 means that it is disabled.

Because the mask changes, the complete network learns parameters that remain useful across many different reduced configurations.

The Keep Probability

Let:\[ p=\text{keep\_prob} \]

Then:\[ P(\text{unit retained})=p \]

and:\[ P(\text{unit removed})=1-p \]

For example, if:\[ p=0.8 \]

then each unit has:

  • An 80% probability of being retained
  • A 20% probability of being removed

A lower keep probability produces stronger regularization.

Creating the Dropout Mask

Suppose dropout is being applied to layer 3. Its activation matrix is:\[ A^{[3]} \]

For vectorized training with \(m\) examples:\[ A^{[3]} \in \mathbb{R}^{n^{[3]}\times m} \]

Create a random matrix with the same shape:

D3 = np.random.rand(*A3.shape)

Every element is sampled independently from a uniform distribution between 0 and 1.

Convert this matrix into a Boolean dropout mask:

D3 = D3 < keep_prob

If keep_prob is 0.8, each element has an 80% probability of being True and a 20% probability of being False.

The shape remains:

assert D3.shape == A3.shape

Applying the Mask

Apply the mask to the activation matrix using element-wise multiplication:

A3 = A3 * D3

The shorter equivalent is:

A3 *= D3

NumPy treats:

  • True as 1
  • False as 0

Therefore, every activation corresponding to False becomes zero.

Mathematically:\[ A^{[3]} := A^{[3]}\odot D^{[3]} \]

where \(\odot\) denotes element-wise multiplication.

Why the Remaining Activations Must Be Scaled

Suppose layer 3 contains 50 units and:\[ \text{keep\_prob}=0.8 \]

On average:\[ 50(1-0.8)=10 \]

units will be disabled, leaving approximately 40 active units.

Without scaling, the average magnitude of the layer’s output would be reduced to approximately 80% of its original value.

The next layer computes:\[ Z^{[4]}=W^{[4]}A^{[3]}+b^{[4]} \]

If 20% of the elements in \(A^{[3]}\) are set to zero, the expected value of \(Z^{[4]}\) also decreases.

To compensate, divide the retained activations by the keep probability:

A3 = A3 / keep_prob

Combining the two steps:

A3 = (A3 * D3) / keep_prob

This procedure is called inverted dropout.

Why Inverted Dropout Preserves the Expected Activation

Consider one activation \(a\) and a random mask \(d\), where:\[ d= \begin{cases} 1 & \text{with probability }p\\ 0 & \text{with probability }1-p \end{cases} \]

Without scaling, the dropped activation is:\[ \tilde{a}=da \]

Its expected value is:\[ \mathbb{E}[\tilde{a}] = \mathbb{E}[d]a = pa \]

The expected value has been reduced by a factor of \(p\).

With inverted dropout:\[ \tilde{a}=\frac{da}{p} \]

Its expected value becomes:\[ \mathbb{E}[\tilde{a}] = \frac{\mathbb{E}[d]a}{p} = \frac{pa}{p} = a \]

Therefore:\[ \mathbb{E}[\tilde{a}]=a \]

Dividing by keep_prob preserves the expected value of the activation.

This is the defining scaling step of inverted dropout.

Complete Forward-Propagation Example

A reusable inverted-dropout function can be written as:

import numpy as np


def apply_inverted_dropout(A, keep_prob):
    """
    Apply inverted dropout to an activation matrix.

    Parameters
    ----------
    A : np.ndarray
        Activation matrix with shape (number_of_units, number_of_examples).
    keep_prob : float
        Probability of retaining each activation.

    Returns
    -------
    A_dropout : np.ndarray
        Scaled activation matrix after dropout.
    D : np.ndarray
        Boolean dropout mask used during this pass.
    """
    D = np.random.rand(*A.shape) < keep_prob
    A_dropout = (A * D) / keep_prob

    return A_dropout, D

It can be applied to layer 3 as follows:

Z3 = np.dot(W3, A2) + b3
A3 = relu(Z3)

A3, D3 = apply_inverted_dropout(A3, keep_prob=0.8)

The mask D3 should be stored because the same mask is needed during backpropagation.

Dropout During Backpropagation

If an activation was disabled during forward propagation, it must remain disabled during the corresponding backward pass.

Suppose dA3 is the derivative flowing backward through layer 3. Apply the same mask:

dA3 = dA3 * D3
dA3 = dA3 / keep_prob

Equivalently:

dA3 = (dA3 * D3) / keep_prob

Mathematically:\[ dA^{[3]} := \frac{dA^{[3]}\odot D^{[3]}}{p} \]

A newly generated mask must not be used during this backward pass. The backward computation must correspond to the same reduced network used during forward propagation.

Reuse the forward-pass mask during the corresponding backward pass.

Different Masks Across Iterations

Even when processing the same data again, dropout should normally generate a new random mask.

For example:

for epoch in range(num_epochs):
    Z3 = np.dot(W3, A2) + b3
    A3 = relu(Z3)

    A3, D3 = apply_inverted_dropout(A3, keep_prob)

Each iteration produces a new D3. The model therefore trains across many different reduced subnetworks instead of repeatedly using the same one.

For vectorized data, different activations can be removed for different examples because the mask has the same shape as the activation matrix.

Do Not Use Dropout During Evaluation

During evaluation or prediction, dropout is disabled. The complete neural network is used:\[ Z^{[1]}=W^{[1]}A^{[0]}+b^{[1]} \]\[ A^{[1]}=g^{[1]}(Z^{[1]}) \]\[ Z^{[2]}=W^{[2]}A^{[1]}+b^{[2]} \]\[ A^{[2]}=g^{[2]}(Z^{[2]}) \]

The process continues until the final prediction:\[ \hat{Y}=A^{[L]} \]

No random mask is generated, and no units are removed.

A simple structure is:

def forward_propagation(X, parameters, training, keep_prob):
    A = X

    for layer in hidden_layers:
        Z = np.dot(layer.W, A) + layer.b
        A = relu(Z)

        if training and keep_prob < 1.0:
            A, D = apply_inverted_dropout(A, keep_prob)

    return A

During training:

predictions = forward_propagation(
    X_train,
    parameters,
    training=True,
    keep_prob=0.8,
)

During evaluation:

predictions = forward_propagation(
    X_dev,
    parameters,
    training=False,
    keep_prob=1.0,
)

Why Random Evaluation Is Undesirable

If dropout were used during evaluation, identical inputs could produce different predictions on different runs.

This would:

  • Add noise to predictions
  • Reduce reproducibility
  • Make evaluation unstable
  • Require repeated computation to estimate an average

In theory, predictions could be generated many times with different masks and then averaged. However, that would be computationally inefficient.

Using the complete network provides a practical approximation to averaging across the many subnetworks encountered during training.

Why No Extra Evaluation-Time Scaling Is Needed

Older versions of dropout sometimes retained activations without dividing by the keep probability during training. Those approaches required additional scaling during evaluation.

Inverted dropout moves the scaling into the training phase:\[ A_{\text{dropout}} = \frac{A\odot D}{p} \]

Because the expected activation is already preserved, evaluation can use the complete network directly.

Inverted dropout simplifies evaluation because no additional activation scaling is required.

A Layer-Level Implementation

The following example applies dropout to selected hidden layers:

def forward_layer(
    A_prev,
    W,
    b,
    activation,
    training=False,
    keep_prob=1.0,
):
    Z = np.dot(W, A_prev) + b
    A = activation(Z)
    D = None

    if training and keep_prob < 1.0:
        D = np.random.rand(*A.shape) < keep_prob
        A = (A * D) / keep_prob

    cache = {
        "A_prev": A_prev,
        "W": W,
        "b": b,
        "Z": Z,
        "D": D,
        "keep_prob": keep_prob,
    }

    return A, cache

The cached mask can then be used during backpropagation:

def apply_dropout_backward(dA, cache):
    D = cache["D"]
    keep_prob = cache["keep_prob"]

    if D is not None:
        dA = (dA * D) / keep_prob

    return dA

Common Implementation Mistakes

Forgetting to scale the activations

Incorrect:

A3 = A3 * D3

Correct inverted dropout:

A3 = (A3 * D3) / keep_prob

Generating a new mask during backpropagation

Incorrect:

D3_backward = np.random.rand(*dA3.shape) < keep_prob
dA3 = dA3 * D3_backward

Correct:

dA3 = (dA3 * D3) / keep_prob

The backward pass must use the mask from the forward pass.

Using dropout during evaluation

Incorrect:

A3, D3 = apply_inverted_dropout(A3, keep_prob=0.8)

during prediction.

Correct:

keep_prob = 1.0

or bypass the dropout operation entirely.

Using inconsistent keep probabilities

The same layer must use the same keep probability during its corresponding forward and backward calculations.

Setting keep_prob outside its valid range

The valid range is:\[ 0<p\leq1 \]

A defensive check can be added:

assert 0 < keep_prob <= 1

Training and Evaluation Summary

StageRandom maskActivation scalingFull network
TrainingYesDivide by keep_probNo
EvaluationNoNo additional scalingYes

Key Takeaway

Inverted dropout uses three operations during training:

D = np.random.rand(*A.shape) < keep_prob
A = A * D
A = A / keep_prob

Mathematically:\[ D^{[l]}\sim\operatorname{Bernoulli}(p) \]\[ A^{[l]} := \frac{A^{[l]}\odot D^{[l]}}{p} \]

The same mask is reused during backpropagation:\[ dA^{[l]} := \frac{dA^{[l]}\odot D^{[l]}}{p} \]

During evaluation, dropout is disabled and the complete network produces the prediction.

Inverted dropout preserves expected activation values during training, allowing evaluation to use the full network without additional scaling.

Similar Posts

Questions, corrections, or additional insights?