Regularization: Reducing Overfitting in Neural Networks

When a model performs well on the training set but poorly on the development set, it has a high-variance problem. In other words, the model is overfitting.

Obtaining more training data is often an effective solution, but additional data may be expensive or unavailable. Regularization provides another reliable way to reduce variance by discouraging the model from relying on excessively large parameter values.

Why Use Regularization?

An overfitted model has often learned details that are specific to the training data rather than patterns that generalize to new examples.

Regularization modifies the cost function so that the model must balance two objectives:

1. Fit the training data well.
2. Keep its parameters reasonably small.

Regularization discourages unnecessary model complexity by adding a penalty for large weights.

The strength of this penalty is controlled by a regularization parameter called \(\lambda\).

L2 Regularization in Logistic Regression

For logistic regression, the original cost function is:\[ J(w,b) = \frac{1}{m} \sum_{i=1}^{m} L\left(\hat{y}^{(i)},y^{(i)}\right) \]

where:

• \(m\) is the number of training examples.
• \(w\in\mathbb{R}^{n_x}\) is the weight vector.
• \(b\in\mathbb{R}\) is the bias.
• \(L\) is the loss for one example.

For binary classification:\[ L(\hat{y},y) = -\left[ y\log\hat{y} + (1-y)\log(1-\hat{y}) \right] \]

L2 regularization adds a penalty based on the squared magnitude of \(w\):\[ J_{\text{reg}}(w,b) = \frac{1}{m} \sum_{i=1}^{m} L\left(\hat{y}^{(i)},y^{(i)}\right) + \frac{\lambda}{2m}\lVert w\rVert_2^2 \]

The squared L2 norm is:\[ \lVert w\rVert_2^2 = \sum_{j=1}^{n_x}w_j^2 \]

It can also be written as:\[ \lVert w\rVert_2^2=w^Tw \]

The complete regularized cost is therefore:\[ J_{\text{reg}}(w,b) = \frac{1}{m} \sum_{i=1}^{m} L\left(\hat{y}^{(i)},y^{(i)}\right) + \frac{\lambda}{2m} \sum_{j=1}^{n_x}w_j^2 \]

The first term measures prediction error. The second penalizes large weights.

Why the Bias Is Usually Not Regularized

It is possible to include a penalty for \(b\), but this is commonly omitted.

The weight vector may contain hundreds, thousands, or millions of parameters, whereas \(b\) is only one number. Most of the model’s complexity therefore comes from \(w\), not from the bias.

Regularizing a single bias value usually has little practical effect compared with regularizing the full weight vector.

Weight parameters are normally regularized, while bias parameters are usually left unregularized.

Understanding the Regularization Parameter

The parameter \(\lambda\) controls how strongly large weights are penalized.

When \(\lambda=0\)

The regularization term disappears:\[ J_{\text{reg}}=J \]

The model minimizes only the original prediction loss. A sufficiently flexible model may overfit.

When \(\lambda\) is moderate

The model is encouraged to fit the data while keeping weights reasonably small. This can reduce variance and improve development-set performance.

When \(\lambda\) is too large

The regularization penalty may dominate the objective. The weights can become so small that the model loses its ability to fit the training data.

This may introduce high bias or underfitting.

The value of \(\lambda\) is therefore selected by comparing performance on development data.

Regularization as a Hyperparameter

Unlike \(w\) and \(b\), \(\lambda\) is not learned directly by gradient descent in the standard formulation. It is a hyperparameter chosen during model development.

A common process is:

1. Select several candidate values of \(\lambda\).
2. Train a model using each value.
3. Compare their development performance.
4. Choose the value that provides the best generalization.

The goal is to balance data fitting and parameter control.

In Python, lambda is a reserved keyword used to define anonymous functions. Code therefore commonly uses a name such as:

lambd

For example:

lambd = 0.01

L1 Regularization

Another option is L1 regularization:\[ J_{\text{reg}}(w,b) = J(w,b) + \frac{\lambda}{m}\lVert w\rVert_1 \]

where:\[ \lVert w\rVert_1 = \sum_{j=1}^{n_x}|w_j| \]

L1 regularization tends to produce sparse parameter vectors, meaning that many components of \(w\) may become exactly zero or close to zero.

A sparse vector has the form:\[ w= \begin{bmatrix} 0\\ w_2\\ 0\\ 0\\ w_5\\ \vdots \end{bmatrix} \]

Sparsity can sometimes help with feature selection or storage. However, it does not automatically produce major practical compression benefits, and L1 regularization is less commonly used than L2 regularization in neural networks.

L1 and L2 Regularization Compared

The constants used in front of either penalty may vary by convention. The important factor is the effective strength of \(\lambda\).

L2 Regularization in a Neural Network

A neural network has a weight matrix and bias vector for every layer:\[ W^{[1]},b^{[1]}, W^{[2]},b^{[2]}, \ldots, W^{[L]},b^{[L]} \]

where \(L\) is the number of layers.

Without regularization, its cost is:\[ J = \frac{1}{m} \sum_{i=1}^{m} L\left(\hat{y}^{(i)},y^{(i)}\right) \]

To apply L2 regularization, add a penalty for every weight matrix:\[ J_{\text{reg}} = \frac{1}{m} \sum_{i=1}^{m} L\left(\hat{y}^{(i)},y^{(i)}\right) + \frac{\lambda}{2m} \sum_{l=1}^{L} \left\lVert W^{[l]}\right\rVert_F^2 \]

Bias vectors are typically excluded from this penalty.

The Frobenius Norm

For a matrix, the sum of its squared elements is called the squared Frobenius norm.

If:\[ W^{[l]} \in \mathbb{R}^{n^{[l]}\times n^{[l-1]}} \]

then:\[ \left\lVert W^{[l]}\right\rVert_F^2 = \sum_{i=1}^{n^{[l]}} \sum_{j=1}^{n^{[l-1]}} \left(W_{ij}^{[l]}\right)^2 \]

The subscript \(F\) indicates the Frobenius norm.

Despite the specialized name, the calculation is straightforward:

Square every element in the matrix and add the results.

In NumPy:

frobenius_squared = np.sum(np.square(W))

The regularization penalty across all layers can be calculated conceptually as:

l2_penalty = 0
for W in weight_matrices:
    l2_penalty += np.sum(np.square(W))
l2_penalty *= lambd / (2 * m)

How Regularization Changes Backpropagation

Without regularization, backpropagation calculates the gradient:\[ dW_{\text{data}}^{[l]} = \frac{\partial J}{\partial W^{[l]}} \]

After adding L2 regularization, the derivative of the penalty contributes:\[ \frac{\lambda}{m}W^{[l]} \]

The new gradient is:\[ dW^{[l]} = dW_{\text{data}}^{[l]} + \frac{\lambda}{m}W^{[l]} \]

The bias gradient remains unchanged:\[ db^{[l]}=db_{\text{data}}^{[l]} \]

because the bias is not included in the regularization term.

The parameter update is then:\[ W^{[l]} := W^{[l]}-\alpha dW^{[l]} \]

Substituting the regularized gradient:\[ W^{[l]} := W^{[l]} – \alpha \left( dW_{\text{data}}^{[l]} + \frac{\lambda}{m}W^{[l]} \right) \]

Why L2 Regularization Is Called Weight Decay

Rearranging the update gives:\[ W^{[l]} := W^{[l]} – \frac{\alpha\lambda}{m}W^{[l]} – \alpha dW_{\text{data}}^{[l]} \]

Factor the weight matrix from the first two terms:\[ W^{[l]} := \left( 1-\frac{\alpha\lambda}{m} \right) W^{[l]} – \alpha dW_{\text{data}}^{[l]} \]

The factor:\[ 1-\frac{\alpha\lambda}{m} \]

is slightly smaller than 1 when \(\alpha\), \(\lambda\), and \(m\) are positive.

At every update, the existing weight matrix is therefore multiplied by a value slightly below 1 before the ordinary gradient update is applied.

This gradually reduces—or decays—the magnitude of the weights.

L2 regularization is called weight decay because each update slightly shrinks the existing weights.

Vectorized Implementation

For one layer, a regularized gradient calculation may look like:

dW = dW_data + (lambd / m) * W
db = db_data
W = W - learning_rate * dW
b = b - learning_rate * db

An equivalent update for \(W\) is:

W = (
    1 - learning_rate * lambd / m
) * W - learning_rate * dW_data

The first form is usually clearer because it directly expresses the derivative of the regularized cost.

How Regularization Addresses Overfitting

An overfitted neural network may use large weights to create highly sensitive decision boundaries. Small changes in the input can then produce large changes in the output.

L2 regularization discourages these large weights. As the parameters become smaller, the network generally becomes less sensitive to isolated or noisy examples.

The model is then encouraged to learn smoother and more broadly useful patterns.

Regularization does not necessarily make the network physically smaller. Instead, it limits the effective influence of its parameters.

Bias–Variance Effect

Regularization primarily targets variance:\[ \text{Stronger regularization} \rightarrow \text{smaller weights} \rightarrow \text{lower variance} \]

However, excessive regularization can increase bias:\[ \text{Excessive regularization} \rightarrow \text{insufficient flexibility} \rightarrow \text{underfitting} \]

This is why \(\lambda\) must be tuned using development performance rather than made as large as possible.

Key Takeaway

L2 regularization modifies the neural-network cost function by penalizing the squared magnitude of its weight matrices:\[ J_{\text{reg}} = J + \frac{\lambda}{2m} \sum_{l=1}^{L} \left\lVert W^{[l]}\right\rVert_F^2 \]

The corresponding gradient becomes:\[ dW^{[l]} = dW_{\text{data}}^{[l]} + \frac{\lambda}{m}W^{[l]} \]

and the update can be written as:\[ W^{[l]} := \left( 1-\frac{\alpha\lambda}{m} \right)W^{[l]} – \alpha dW_{\text{data}}^{[l]} \]

Regularization reduces overfitting by discouraging unnecessarily large weights.

L2 regularization is the most common form used in neural networks. Its strength is controlled by \(\lambda\), which should be chosen to reduce variance without introducing excessive bias.