Mini-Batch Gradient Descent
Training a neural network is an iterative process. Multiple architectures and hyperparameter combinations may need to be evaluated before finding a model that performs well.
This becomes expensive when the training set contains millions of examples. Batch gradient descent processes the complete dataset before making a single parameter update. Mini-batch gradient descent accelerates learning by dividing the data into smaller subsets and updating the parameters after processing each one.
The Limitation of Batch Gradient Descent
Suppose the complete training set contains \(m\) examples arranged as columns:\[ X= \begin{bmatrix} x^{(1)} & x^{(2)} & \cdots & x^{(m)} \end{bmatrix} \]
For \(n_x\) input features:\[ X\in\mathbb{R}^{n_x\times m} \]
The labels are:\[ Y= \begin{bmatrix} y^{(1)} & y^{(2)} & \cdots & y^{(m)} \end{bmatrix} \]
For binary classification:\[ Y\in\mathbb{R}^{1\times m} \]
Vectorization allows all \(m\) examples to be processed efficiently. However, if \(m\) is extremely large—such as 5 million or 50 million—even a vectorized computation across the complete dataset can take substantial time.
With batch gradient descent, the model must process all \(m\) examples before performing one update:\[ W^{[l]} := W^{[l]}-\alpha dW^{[l]} \]\[ b^{[l]} := b^{[l]}-\alpha db^{[l]} \]
It must then process the entire dataset again before taking another step.
Batch gradient descent makes only one parameter update after a complete pass through the training set.
Dividing the Dataset into Mini-Batches
Mini-batch gradient descent divides the complete dataset into smaller subsets called mini-batches.
Suppose:\[ m=5{,}000{,}000 \]
and each mini-batch contains:\[ m_{\text{batch}}=1{,}000 \]
examples.
The number of mini-batches is:\[ \frac{5{,}000{,}000}{1{,}000} = 5{,}000 \]
The first mini-batch contains:\[ x^{(1)},x^{(2)},\ldots,x^{(1000)} \]
The second contains:\[ x^{(1001)},x^{(1002)},\ldots,x^{(2000)} \]
This continues until all examples have been included.
Mini-Batch Notation
Curly braces identify individual mini-batches:\[ X^{\{1\}},Y^{\{1\}} \]\[ X^{\{2\}},Y^{\{2\}} \]\[ \vdots \]\[ X^{\{5000\}},Y^{\{5000\}} \]
The \(t\)-th mini-batch is:\[ \left( X^{\{t\}}, Y^{\{t\}} \right) \]
Three different index styles now have distinct meanings.
Parentheses identify examples
\[ x^{(i)} \]
means example \(i\).
Square brackets identify layers
\[ Z^{[l]} \]
means the linear value associated with layer \(l\).
Curly braces identify mini-batches
\[ X^{\{t\}} \]
means input data from mini-batch \(t\).
Mini-Batch Dimensions
If each mini-batch contains 1,000 examples, then:\[ X^{\{t\}} \in \mathbb{R}^{n_x\times1000} \]
For binary labels:\[ Y^{\{t\}} \in \mathbb{R}^{1\times1000} \]
Each column still represents one example. Mini-batching changes the number of columns processed at once, not the feature layout.
More generally, for batch size \(m_{\text{batch}}\):\[ X^{\{t\}} \in \mathbb{R}^{n_x\times m_{\text{batch}}} \]\[ Y^{\{t\}} \in \mathbb{R}^{n_y\times m_{\text{batch}}} \]
The final mini-batch may be smaller if \(m\) is not divisible by the chosen batch size.
Batch Gradient Descent and Mini-Batch Gradient Descent
The word “batch” refers to the collection of examples used to calculate one gradient update.
Batch gradient descent
Uses the complete training set:\[ (X,Y) \]
One complete pass through the data produces one update.
Mini-batch gradient descent
Uses one smaller subset at a time:\[ \left( X^{\{t\}}, Y^{\{t\}} \right) \]
Each mini-batch produces one update.
Mini-batch gradient descent begins improving the parameters before the model has processed the entire dataset.
Forward Propagation on One Mini-Batch
For mini-batch \(t\), use:\[ A^{[0]}=X^{\{t\}} \]
The first layer computes:\[ Z^{[1]} = W^{[1]}X^{\{t\}}+b^{[1]} \]\[ A^{[1]} = g^{[1]}\left(Z^{[1]}\right) \]
For later layers:\[ Z^{[l]} = W^{[l]}A^{[l-1]}+b^{[l]} \]\[ A^{[l]} = g^{[l]}\left(Z^{[l]}\right) \]
Continue until the final output:\[ A^{[L]}=\hat{Y}^{\{t\}} \]
These operations remain vectorized. Instead of processing all 5 million examples at once, the network processes 1,000 examples together.
Cost for One Mini-Batch
For mini-batch \(t\), the data cost is:\[ J^{\{t\}} = \frac{1}{m_{\text{batch}}} \sum_{i=1}^{m_{\text{batch}}} L\left( \hat{y}^{\{t\}(i)}, y^{\{t\}(i)} \right) \]
Here, \(i\) identifies an example within the current mini-batch.
If L2 regularization is included:\[ J_{\text{reg}}^{\{t\}} = J^{\{t\}} + \frac{\lambda}{2m_{\text{batch}}} \sum_{l=1}^{L} \left\lVert W^{[l]}\right\rVert_F^2 \]
Backpropagation then calculates the derivatives of this mini-batch cost.
Updating the Parameters
After backpropagation on mini-batch \(t\), update every layer:\[ W^{[l]} := W^{[l]}-\alpha dW^{[l]} \]\[ b^{[l]} := b^{[l]}-\alpha db^{[l]} \]
The process then moves to the next mini-batch.
For 5,000 mini-batches:\[ t=1,2,\ldots,5000 \]
the model performs 5,000 parameter updates during one pass through the complete dataset.
What Is an Epoch?
An epoch is one complete pass through the training set.
With batch gradient descent:\[ 1\text{ epoch} = 1\text{ gradient update} \]
With 5,000 mini-batches:\[ 1\text{ epoch} = 5000\text{ gradient updates} \]
This is a major reason mini-batch gradient descent can make progress more quickly.
An epoch describes how much data has been processed, not how many updates have been made.
One Epoch in Python
A basic implementation is:
for X_batch, Y_batch in mini_batches:
activations, caches = forward_propagation(
X_batch,
parameters,
)
cost = compute_cost(
activations[-1],
Y_batch,
)
gradients = backward_propagation(
activations,
Y_batch,
caches,
)
parameters = update_parameters(
parameters,
gradients,
learning_rate,
)Each pass through mini_batches represents one epoch.
Training for Multiple Epochs
Neural networks normally require multiple passes through the dataset:
for epoch in range(num_epochs):
mini_batches = create_mini_batches(
X_train,
Y_train,
batch_size,
shuffle=True,
)
for X_batch, Y_batch in mini_batches:
activations, caches = forward_propagation(
X_batch,
parameters,
)
gradients = backward_propagation(
activations,
Y_batch,
caches,
)
parameters = update_parameters(
parameters,
gradients,
learning_rate,
)Training continues until the model converges sufficiently or another stopping condition is reached.
Shuffling Before Creating Mini-Batches
The training examples should normally be shuffled before each epoch.
Without shuffling, examples may be grouped by:
- Class
- Collection time
- Geographic region
- Data source
- Another systematic ordering
This could make individual mini-batches unrepresentative of the overall training distribution.
A consistent shuffle must be applied to both \(X\) and \(Y\):
permutation = np.random.permutation(
X.shape[1]
)
X_shuffled = X[:, permutation]
Y_shuffled = Y[:, permutation]Then divide the shuffled data into mini-batches.
Creating Mini-Batches
import numpy as np
def create_mini_batches(
X,
Y,
batch_size,
shuffle=True,
):
m = X.shape[1]
if shuffle:
permutation = np.random.permutation(m)
X = X[:, permutation]
Y = Y[:, permutation]
mini_batches = []
for start in range(0, m, batch_size):
end = min(start + batch_size, m)
X_batch = X[:, start:end]
Y_batch = Y[:, start:end]
mini_batches.append(
(X_batch, Y_batch)
)
return mini_batchesIf the final mini-batch contains fewer examples, its cost and gradients should use its actual size:
m_batch = X_batch.shape[1]rather than assuming every mini-batch has the full requested size.
Why Mini-Batches Speed Up Training
Mini-batch gradient descent combines two advantages.
It updates more frequently
The model does not wait for the entire dataset before making progress.
With 5 million examples and a mini-batch size of 1,000, it performs 5,000 updates per epoch rather than one.
It remains vectorized
Each mini-batch contains enough examples to use efficient matrix operations:\[ Z^{[l]} = W^{[l]}A^{[l-1]}+b^{[l]} \]
The computation can take advantage of optimized CPU or GPU operations.
Mini-Batch Gradients Are Estimates
A mini-batch gradient is not exactly the same as the gradient calculated from the entire dataset.
The full gradient is:\[ \nabla J = \frac{1}{m} \sum_{i=1}^{m} \nabla L^{(i)} \]
A mini-batch gradient is:\[ \nabla J^{\{t\}} = \frac{1}{m_{\text{batch}}} \sum_{i\in\text{batch }t} \nabla L^{(i)} \]
If the mini-batch is representative, its gradient provides a useful estimate of the full gradient.
Because the estimate varies from batch to batch, the optimization path is noisier than batch gradient descent. However, the increased update frequency often produces much faster practical progress.
Comparison of Gradient-Descent Variants
| Method | Examples per update | Updates per epoch | Main characteristic |
|---|---|---|---|
| Batch gradient descent | All \(m\) examples | 1 | Stable but slow on large datasets |
| Mini-batch gradient descent | A subset | Multiple | Efficient and widely used |
| Stochastic gradient descent | 1 example | \(m\) | Very noisy and less vectorized |
Mini-batch gradient descent provides a practical balance between stable gradients and efficient computation.
The Importance of Batch Size
The mini-batch size controls the trade-off between:
- Gradient noise
- Memory usage
- Update frequency
- Hardware efficiency
A larger mini-batch produces a gradient closer to the full-batch gradient but requires more memory and provides fewer updates per epoch.
A smaller mini-batch provides more frequent updates but introduces greater noise and may use hardware less efficiently.
The best size depends on:
- Dataset size
- Model architecture
- Available memory
- CPU or GPU characteristics
- Optimization algorithm
Key Takeaway
Mini-batch gradient descent divides the training data into smaller subsets:\[ \left( X^{\{1\}},Y^{\{1\}} \right), \left( X^{\{2\}},Y^{\{2\}} \right), \ldots \]
For each mini-batch, the network performs:\[ \text{forward propagation} \rightarrow \text{cost calculation} \rightarrow \text{backpropagation} \rightarrow \text{parameter update} \]
Mini-batch gradient descent combines frequent parameter updates with efficient vectorized computation.
With large datasets, it usually trains much faster than processing the complete dataset before every update, which is why it is the standard approach for modern neural-network optimization.