Normalizing Inputs to Speed Up Neural Network Training

ByJu Yeon Eum

Input normalization is a simple preprocessing technique that can make neural-network training significantly faster. It transforms input features so that they have approximately zero mean and similar variance.

When features use dramatically different scales, the cost function can become difficult to optimize. Normalization gives gradient descent a better-conditioned optimization landscape, allowing it to reach the minimum more directly.

The Problem with Different Feature Scales

Suppose a dataset has two input features:\[ x= \begin{bmatrix} x_1\\ x_2 \end{bmatrix} \]

and their ranges are:\[ 1\leq x_1\leq1000 \]\[ 0\leq x_2\leq1 \]

The first feature varies across a range of approximately 999, while the second varies across a range of only 1.

This difference affects the geometry of the cost function and can make gradient descent inefficient.

Input normalization places features on comparable scales so that optimization can progress more evenly in every parameter direction.

The Two Normalization Steps

Input normalization usually consists of:

  1. Subtracting the mean
  2. Dividing by the standard deviation

These operations are performed separately for each feature.

Step 1: Subtract the Mean

Suppose the training set contains \(m\) examples:\[ x^{(1)},x^{(2)},\ldots,x^{(m)} \]

Calculate the mean vector:\[ \mu = \frac{1}{m} \sum_{i=1}^{m}x^{(i)} \]

Because each example contains multiple features, \(\mu\) is also a vector:\[ \mu= \begin{bmatrix} \mu_1\\ \mu_2\\ \vdots\\ \mu_{n_x} \end{bmatrix} \]

Subtract this mean from every example:\[ x_{\text{centered}}^{(i)} = x^{(i)}-\mu \]

After this transformation, each feature has mean approximately zero.

Geometrically, mean subtraction shifts the complete dataset so that it is centered around the origin.

Step 2: Normalize the Variance

After centering the data, calculate the variance of every feature:\[ \sigma^2 = \frac{1}{m} \sum_{i=1}^{m} \left( x_{\text{centered}}^{(i)} \right)^2 \]

The square is applied element-wise, so \(\sigma^2\) is a vector containing one variance for every feature:\[ \sigma^2= \begin{bmatrix} \sigma_1^2\\ \sigma_2^2\\ \vdots\\ \sigma_{n_x}^2 \end{bmatrix} \]

The corresponding standard-deviation vector is:\[ \sigma=\sqrt{\sigma^2} \]

Normalize every centered example:\[ x_{\text{norm}}^{(i)} = \frac{x_{\text{centered}}^{(i)}}{\sigma} \]

The division is performed element-wise.

Combining both steps:\[ x_{\text{norm}}^{(i)} = \frac{x^{(i)}-\mu}{\sigma} \]

After normalization, each feature should have approximately:\[ \text{mean}=0 \]

and:\[ \text{variance}=1 \]

Vectorized Normalization

If examples are stored as columns:\[ X\in\mathbb{R}^{n_x\times m} \]

then each row represents one feature and each column represents one example.

The normalization can be implemented as:

import numpy as np

mu = np.mean(X_train, axis=1, keepdims=True)
variance = np.mean(
    np.square(X_train - mu),
    axis=1,
    keepdims=True,
)
sigma = np.sqrt(variance)

X_train_norm = (X_train - mu) / sigma

The shapes are:

assert mu.shape == (X_train.shape[0], 1)
assert sigma.shape == (X_train.shape[0], 1)
assert X_train_norm.shape == X_train.shape

NumPy broadcasting applies each feature’s mean and standard deviation across all examples.

Handling Zero or Extremely Small Variance

A feature may have zero variance if it has the same value in every example. Dividing by its standard deviation would cause division by zero.

A small constant \(\varepsilon\) is commonly added for numerical stability:\[ x_{\text{norm}}^{(i)} = \frac{x^{(i)}-\mu}{\sigma+\varepsilon} \]

For example:

epsilon = 1e-8
X_train_norm = (X_train - mu) / (sigma + epsilon)

If a feature is completely constant, it contains no useful information for distinguishing examples and may also be removed during preprocessing.

Use Training Statistics for Every Dataset

The mean and standard deviation must be calculated from the training set:\[ \mu_{\text{train}} = \frac{1}{m_{\text{train}}} \sum_i x_{\text{train}}^{(i)} \]\[ \sigma_{\text{train}}^2 = \frac{1}{m_{\text{train}}} \sum_i \left( x_{\text{train}}^{(i)}-\mu_{\text{train}} \right)^2 \]

Use these same values to transform development, test, and future production data:\[ X_{\text{dev,norm}} = \frac{X_{\text{dev}}-\mu_{\text{train}}} {\sigma_{\text{train}}+\varepsilon} \]\[ X_{\text{test,norm}} = \frac{X_{\text{test}}-\mu_{\text{train}}} {\sigma_{\text{train}}+\varepsilon} \]

In Python:

epsilon = 1e-8

X_dev_norm = (X_dev - mu) / (sigma + epsilon)
X_test_norm = (X_test - mu) / (sigma + epsilon)

Calculate normalization statistics from the training data once, then reuse them everywhere.

Why Development and Test Statistics Should Not Be Estimated Separately

If each dataset is normalized using its own statistics, each one undergoes a different transformation.

That creates an inconsistency:\[ T_{\text{train}}(x) \neq T_{\text{test}}(x) \]

The model is trained on inputs processed by \(T_{\text{train}}\), so evaluation data should pass through that same transformation.

Using test-set statistics can also leak information about the evaluation distribution into preprocessing.

The correct procedure is:

  1. Fit the normalization transformation using training data.
  2. Save \(\mu_{\text{train}}\) and \(\sigma_{\text{train}}\).
  3. Apply the saved transformation to every later input.

A Reusable Normalization Implementation

import numpy as np


def fit_normalizer(X, epsilon=1e-8):
    mu = np.mean(X, axis=1, keepdims=True)
    variance = np.mean(
        np.square(X - mu),
        axis=1,
        keepdims=True,
    )
    sigma = np.sqrt(variance)

    return {
        "mu": mu,
        "sigma": sigma,
        "epsilon": epsilon,
    }


def transform_inputs(X, normalizer):
    mu = normalizer["mu"]
    sigma = normalizer["sigma"]
    epsilon = normalizer["epsilon"]

    return (X - mu) / (sigma + epsilon)

Usage:

normalizer = fit_normalizer(X_train)

X_train_norm = transform_inputs(X_train, normalizer)
X_dev_norm = transform_inputs(X_dev, normalizer)
X_test_norm = transform_inputs(X_test, normalizer)

Why Normalization Helps Gradient Descent

Consider a cost function with two parameters, \(w_1\) and \(w_2\).

When the corresponding input features use dramatically different scales, the cost contours may be highly elongated:\[ J(w_1,w_2) \]

has a narrow, stretched shape rather than a balanced, rounded shape.

Gradient descent follows the direction of steepest descent. In a narrow optimization landscape, that direction may repeatedly point across the valley instead of directly toward the minimum.

The trajectory can oscillate:\[ \text{one side} \rightarrow \text{other side} \rightarrow \text{one side} \rightarrow \cdots \rightarrow \text{minimum} \]

This often requires a small learning rate to avoid overshooting.

Optimization After Normalization

When the input features have comparable scales, the contours of the cost function tend to become more symmetric.

Gradient descent can then follow a more direct path:\[ \text{starting point} \rightarrow \text{minimum} \]

This can provide several benefits:

  • Less oscillation
  • Faster convergence
  • More stable updates
  • Greater tolerance for a larger learning rate
  • Easier hyperparameter selection

Normalization does not change the fundamental task; it changes the geometry of the optimization problem.

Why Feature Scale Affects Parameter Scale

For a linear component:\[ z=w_1x_1+w_2x_2+b \]

suppose:\[ x_1\in[1,1000] \]

and:\[ x_2\in[0,1] \]

To produce contributions of comparable magnitude, \(w_1\) may need to be much smaller than \(w_2\).

For example:\[ w_1\approx0.001 \]

while:\[ w_2\approx1 \]

This creates very different sensitivities across parameter directions and contributes to an elongated cost surface.

After normalization, both features vary on similar scales, allowing their associated parameter directions to behave more similarly during optimization.

When Normalization Matters Most

Normalization is especially important when feature ranges differ dramatically.

For example:\[ x_1\in[0,1] \]\[ x_2\in[1,1000] \]\[ x_3\in[-100,100] \]

These features should generally be normalized before training.

By contrast, suppose:\[ x_1\in[0,1] \]\[ x_2\in[-1,1] \]\[ x_3\in[1,2] \]

These ranges are already reasonably similar. Normalization may provide less improvement, although it is still usually safe and can make preprocessing more consistent.

Normalization Does Not Guarantee an Exact Range

Standardization produces approximately zero mean and unit variance, but it does not guarantee that every value lies between \(-1\) and \(1\).

An extreme observation may still become:\[ x_{\text{norm}}=3.7 \]

or:\[ x_{\text{norm}}=-4.2 \]

The objective is not to force all observations into a fixed interval. The objective is to give features comparable centers and scales.

Normalization and Model Performance

Normalization primarily improves optimization. It does not necessarily change the best function that the network could theoretically represent.

However, faster and more stable optimization can indirectly improve practical results because:

  • Training reaches a good solution sooner.
  • A poorly chosen learning rate is less damaging.
  • Parameter updates become better balanced.
  • Deep networks become easier to train.

It is therefore a standard preprocessing step for many numerical datasets.

Common Mistakes

Using different statistics for each dataset

Incorrect:

train_mu = np.mean(X_train, axis=1, keepdims=True)
test_mu = np.mean(X_test, axis=1, keepdims=True)

Correct:

mu = np.mean(X_train, axis=1, keepdims=True)

X_train_centered = X_train - mu
X_test_centered = X_test - mu

Forgetting which axis contains examples

If examples are columns, compute statistics across columns:

mu = np.mean(X_train, axis=1, keepdims=True)

If examples are rows, the appropriate axis would be different. Always confirm the data layout before normalizing.

Dividing by variance instead of standard deviation

The variance is:\[ \sigma^2 \]

but standard normalization divides by:\[ \sigma=\sqrt{\sigma^2} \]

Correct:

sigma = np.sqrt(variance)
X_norm = (X - mu) / (sigma + epsilon)

Recomputing statistics in production

Future inputs should use the saved training values of \(\mu\) and \(\sigma\). Recomputing them for every batch can create inconsistent predictions.

Key Takeaway

Input normalization transforms each feature using:\[ x_{\text{norm}} = \frac{x-\mu_{\text{train}}} {\sigma_{\text{train}}+\varepsilon} \]

where the mean and standard deviation are calculated only from the training data.

Apply the same training-derived normalization transformation to development, test, and future inputs.

Normalization places features on comparable scales, makes the cost surface better conditioned, reduces oscillation during gradient descent, and often allows the model to train faster with a larger and more stable learning rate.

Similar Posts

Leave a Reply