Practical Tips for Gradient Checking

ByJu Yeon Eum

Gradient checking is a debugging technique used to verify that backpropagation computes correct derivatives. It compares analytical gradients from backpropagation with numerical approximations obtained by slightly perturbing each parameter.

Because numerical gradient checking is computationally expensive, it should be used only during debugging—not during normal training.

A Quick Review of Gradient Checking

Suppose all neural-network parameters are reshaped and concatenated into one vector:\[ \theta \]

Similarly, combine all gradients calculated through backpropagation into:\[ d\theta \]

For each component \(\theta_i\), calculate a numerical approximation:\[ d\theta_{\text{approx},i} = \frac{ J(\theta_1,\ldots,\theta_i+\varepsilon,\ldots) – J(\theta_1,\ldots,\theta_i-\varepsilon,\ldots) }{ 2\varepsilon } \]

This is the two-sided numerical derivative.

A typical value is:\[ \varepsilon=10^{-7} \]

The complete numerical gradient vector is:\[ d\theta_{\text{approx}} = \begin{bmatrix} d\theta_{\text{approx},1}\\ d\theta_{\text{approx},2}\\ \vdots\\ d\theta_{\text{approx},n} \end{bmatrix} \]

It is then compared with the analytical gradient \(d\theta\).

Measuring the Difference

A useful normalized difference is:\[ \text{difference} = \frac{ \left\lVert d\theta-d\theta_{\text{approx}} \right\rVert_2 }{ \left\lVert d\theta\right\rVert_2 + \left\lVert d\theta_{\text{approx}}\right\rVert_2 } \]

This relative measurement is more informative than examining the absolute difference alone because it accounts for the overall gradient scale.

A conceptual implementation is:

numerator = np.linalg.norm(
    gradients - gradients_approx
)

denominator = (
    np.linalg.norm(gradients)
    + np.linalg.norm(gradients_approx)
)

difference = numerator / denominator

A very small result suggests that backpropagation is probably correct. A larger result indicates that some derivatives may be wrong.

The exact threshold depends on numerical precision and the operations in the model, but values near \(10^{-7}\) are generally reassuring for smooth computations implemented in double precision.

Do Not Run Gradient Checking During Training

Calculating one numerical derivative requires evaluating the cost twice:\[ J(\theta_i+\varepsilon) \]

and:\[ J(\theta_i-\varepsilon) \]

If the model has \(n\) parameters, computing the complete numerical gradient requires approximately:\[ 2n \]

forward evaluations.

Modern neural networks may contain millions of parameters, making this process extremely slow.

Backpropagation, by contrast, calculates all derivatives efficiently in one backward pass.

Use backpropagation for training and numerical gradient checking only for debugging.

The intended workflow is:

  1. Implement forward propagation.
  2. Implement backpropagation.
  3. Run gradient checking on a small configuration.
  4. Correct any derivative errors.
  5. Disable gradient checking.
  6. Train normally using backpropagation.

Inspect Individual Components When the Check Fails

A single difference value tells you that something is wrong, but it does not identify the location of the problem.

When gradient checking fails, compare individual components:\[ d\theta_i \quad\text{and}\quad d\theta_{\text{approx},i} \]

Look for indices where the disagreement is especially large.

component_errors = np.abs(
    gradients - gradients_approx
)

largest_indices = np.argsort(
    component_errors
)[-10:]

You can inspect these values:

for index in largest_indices:
    print(
        index,
        gradients[index],
        gradients_approx[index],
        component_errors[index],
    )

The failing indices can often be mapped back to specific parameter groups.

Map Errors Back to Their Parameters

The vector \(\theta\) contains parameters from every layer:\[ W^{[1]},b^{[1]}, W^{[2]},b^{[2]}, \ldots, W^{[L]},b^{[L]} \]

Similarly, \(d\theta\) contains:\[ dW^{[1]},db^{[1]}, dW^{[2]},db^{[2]}, \ldots, dW^{[L]},db^{[L]} \]

Suppose most incorrect components correspond to:\[ db^{[3]} \]

while the components of \(dW^{[3]}\) are accurate. The likely problem is then in the calculation of the layer-3 bias gradient.

Possible causes include:

  • Summing across the wrong axis
  • Forgetting the factor \(1/m\)
  • Returning the wrong shape
  • Incorrect broadcasting
  • Omitting keepdims=True

If the errors correspond mainly to:\[ dW^{[2]} \]

investigate the weight-gradient calculation for layer 2.

The locations of the mismatched components can narrow the search to a specific layer or parameter type.

This method may not reveal the exact mistake immediately, but it provides a useful starting point.

Preserve Parameter Metadata

When flattening parameters into \(\theta\), retain enough metadata to reconstruct their original names and shapes.

For example:

parameter_info = [
    ("W1", W1.shape, W1.size),
    ("b1", b1.shape, b1.size),
    ("W2", W2.shape, W2.size),
    ("b2", b2.shape, b2.size),
]

This makes it possible to determine whether a failing numerical component belongs to W1, b1, W2, or another parameter.

Without this mapping, gradient checking may reveal an incorrect index without showing which part of the model it represents.

Include Regularization in the Cost

If the model uses L2 regularization, gradient checking must use the complete regularized cost:\[ J_{\text{reg}} = J_{\text{data}} + \frac{\lambda}{2m} \sum_{l=1}^{L} \left\lVert W^{[l]}\right\rVert_F^2 \]

The corresponding analytical weight gradient is:\[ dW^{[l]} = dW_{\text{data}}^{[l]} + \frac{\lambda}{m}W^{[l]} \]

If the numerical gradient uses the regularized cost but backpropagation omits the regularization derivative, the check will fail.

The reverse is also true: if backpropagation includes the regularization term but the numerical cost does not, the gradients will disagree.

The numerical and analytical calculations must differentiate exactly the same cost function.

Bias vectors are normally excluded from L2 regularization, so their derivatives do not receive the additional term.

A Regularized Cost Example

def compute_regularized_cost(
    data_cost,
    weights,
    lambd,
    m,
):
    squared_norm = 0.0

    for W in weights:
        squared_norm += np.sum(np.square(W))

    regularization_cost = (
        lambd / (2 * m)
    ) * squared_norm

    return data_cost + regularization_cost

The analytical gradient must then include:

dW = dW_data + (lambd / m) * W

Gradient Checking Does Not Work Directly with Random Dropout

Dropout randomly removes different units during training. If two cost evaluations use different masks, they are evaluating different subnetworks.

The numerical approximation:\[ \frac{ J(\theta_i+\varepsilon) – J(\theta_i-\varepsilon) }{ 2\varepsilon } \]

is meaningful only when both cost values are calculated using the same deterministic function.

With normal dropout:

  • The first evaluation may retain one set of units.
  • The second evaluation may retain another set.
  • Their difference includes random mask variation.
  • The result no longer isolates the effect of changing \(\theta_i\).

Dropout can be viewed theoretically as optimizing an expectation over many possible subnetworks, but calculating that exact objective would require considering an exponentially large number of masks.

Disable Dropout Before Checking Gradients

The simplest solution is to turn off dropout:

keep_prob = 1.0

Then:

  1. Run forward propagation without dropout.
  2. Run backpropagation without dropout.
  3. Verify the derivatives numerically.
  4. Restore the desired keep probabilities afterward.

Disable stochastic behavior before running gradient checking.

This verifies that the main forward and backward calculations are correct. Dropout-specific logic must then be tested separately.

An Alternative: Fix the Dropout Mask

A more advanced option is to generate a dropout mask once and reuse exactly that mask for every numerical evaluation.

This makes the function deterministic:

fixed_mask = (
    np.random.rand(*A.shape) < keep_prob
)

Every calculation of:\[ J(\theta_i+\varepsilon) \]

and:\[ J(\theta_i-\varepsilon) \]

must use this same mask.

Although this can help validate dropout-specific derivatives, it increases implementation complexity. In many cases, disabling dropout provides a simpler and more practical check.

Run the Check Near Initialization

Gradient checking is commonly performed immediately after random initialization.

At this point:\[ W^{[l]} \]

contains small random values and:\[ b^{[l]} \]

is often initialized to zero.

This is useful for catching many common mistakes, including:

  • Incorrect matrix multiplication
  • Wrong transposes
  • Missing averaging factors
  • Incorrect activation derivatives
  • Improper bias gradients
  • Missing regularization terms

However, passing at initialization does not guarantee that every numerical situation is handled correctly.

Run It Again After Some Training

A rare bug may appear only after the parameters move away from their initial values.

For example, an implementation could behave correctly while:\[ W\approx0,\qquad b\approx0 \]

but become inaccurate when:

  • Weights grow
  • Bias values become nonzero
  • Activations enter different regions
  • Piecewise activation boundaries are encountered
  • Numerical stability becomes more important

A stronger debugging process is:

  1. Run gradient checking near initialization.
  2. Train for a limited number of iterations.
  3. Save the resulting parameters.
  4. Run gradient checking again at those parameter values.

Checking gradients at more than one point can reveal bugs hidden by the special conditions near initialization.

This second check is not always necessary, but it can be valuable when the implementation behaves strangely later in training.

Use Small Models and Small Batches

Because numerical checking is expensive, perform it on a reduced configuration:

  • A small number of layers
  • Few hidden units
  • A small batch of examples
  • A deterministic forward pass
  • Double-precision values where possible

The purpose is to validate the mathematical implementation, not to evaluate the full-scale model.

For example:

X_check = X_train[:, :4]
Y_check = Y_train[:, :4]

A small subset can make the process much faster while still exposing derivative errors.

Avoid Non-Differentiable Points

Gradient checking uses a finite approximation around \(\theta_i\). Functions such as ReLU are not differentiable at exactly zero:\[ \operatorname{ReLU}(z)=\max(0,z) \]

If a perturbation crosses the point \(z=0\), the numerical derivative may disagree with the analytical convention even if the implementation is otherwise correct.

Possible responses include:

  • Use a very small \(\varepsilon\).
  • Check whether the disagreement is isolated near a ReLU boundary.
  • Repeat the check at slightly different parameter values.
  • Focus on the overall pattern of errors rather than one boundary component.

A Practical Checklist

Before running gradient checking:

  • Use a small deterministic model.
  • Disable dropout and other random operations.
  • Confirm that parameters and gradients are flattened in the same order.
  • Include all terms in the cost, including regularization.
  • Use the same data for analytical and numerical calculations.
  • Use double precision where possible.
  • Choose a suitably small \(\varepsilon\).

If the check fails:

  • Inspect individual components.
  • Map failing indices back to parameter names.
  • Check whether errors cluster in a specific layer.
  • Verify matrix shapes and transposes.
  • Verify averaging by \(m\).
  • Verify activation derivatives.
  • Verify regularization derivatives.
  • Check for non-differentiable boundaries.

After the check passes:

  • Disable numerical checking.
  • Re-enable dropout if needed.
  • Restore the full model and dataset.
  • Continue using backpropagation alone.

Key Takeaway

Gradient checking compares backpropagation with a numerical approximation:\[ d\theta_{\text{approx},i} = \frac{ J(\theta_i+\varepsilon) – J(\theta_i-\varepsilon) }{ 2\varepsilon } \]

It is a powerful debugging tool, but it is too slow for normal training.

Use gradient checking only to validate backpropagation, then turn it off.

When the check fails, inspect individual components and map them back to their original weights and biases. Include regularization in both the cost and analytical gradients, disable dropout or make it deterministic, and consider checking both near initialization and after the parameters have changed.

Similar Posts

Questions, corrections, or additional insights?