Understanding Mini-Batch Gradient Descent

Mini-batch gradient descent combines the computational efficiency of vectorization with the ability to update a model before processing the complete training set.

Its behavior lies between two extremes:

• Batch gradient descent, which uses every training example for one update
• Stochastic gradient descent, which uses only one example per update

Understanding these three approaches helps explain why intermediate mini-batch sizes are usually preferred.

Cost Behavior with Batch Gradient Descent

Batch gradient descent calculates each update using the complete training set:\[ (X,Y) \]

With full-batch training, every iteration evaluates the same objective:\[ J(W,b) = \frac{1}{m} \sum_{i=1}^{m} L\left( \hat{y}^{(i)}, y^{(i)} \right) \]

If the implementation is correct and the learning rate is appropriate, the cost should generally decrease with each update:\[ J^{(1)} \geq J^{(2)} \geq J^{(3)} \geq \cdots \]

If the cost repeatedly increases, possible causes include:

• A learning rate that is too large
• An incorrect gradient calculation
• A sign error in the update
• An inconsistent cost implementation
• Numerical instability

For deterministic full-batch gradient descent, a smooth downward curve is an important debugging signal.

Cost Behavior with Mini-Batches

Mini-batch gradient descent uses a different subset of data during each update:\[ \left( X^{\{1\}},Y^{\{1\}} \right), \left( X^{\{2\}},Y^{\{2\}} \right), \ldots \]

The cost for batch \(t\) is:\[ J^{\{t\}} = \frac{1}{m_t} \sum_{i=1}^{m_t} L\left( \hat{y}^{\{t\}(i)}, y^{\{t\}(i)} \right) \]

where \(m_t\) is the number of examples in that mini-batch.

Because each update uses different data, these costs are not measurements of exactly the same sample.

One mini-batch may contain relatively easy examples and produce a low cost. The next may contain difficult or mislabeled examples and produce a higher cost.

Therefore, the cost curve may fluctuate:\[ J^{\{1\}} < J^{\{2\}} > J^{\{3\}} < J^{\{4\}} \]

even when training is working correctly.

With mini-batch gradient descent, the cost does not need to decrease after every update, but its overall trend should move downward.

Monitoring a Noisy Cost Curve

Raw mini-batch costs can be difficult to interpret. Useful monitoring options include:

• Plotting a moving average
• Plotting average cost per epoch
• Evaluating a fixed development set periodically
• Comparing the trend over many updates rather than adjacent values

For example:

epoch_cost = 0.0
examples_seen = 0

for X_batch, Y_batch in mini_batches:
    batch_size = X_batch.shape[1]

    batch_cost = train_on_batch(
        X_batch,
        Y_batch,
    )

    epoch_cost += batch_cost * batch_size
    examples_seen += batch_size

epoch_cost /= examples_seen

epoch_cost = 0.0
examples_seen = 0

for X_batch, Y_batch in mini_batches:
    batch_size = X_batch.shape[1]

    batch_cost = train_on_batch(
        X_batch,
        Y_batch,
    )

    epoch_cost += batch_cost * batch_size
    examples_seen += batch_size

epoch_cost /= examples_seen

Weighting each batch cost by its actual size is important when the final mini-batch is smaller.

The First Extreme: Batch Gradient Descent

If the mini-batch size equals the complete training-set size:\[ m_{\text{batch}}=m \]

then there is only one mini-batch:\[ X^{\{1\}}=X \]\[ Y^{\{1\}}=Y \]

This is batch gradient descent.

Advantages

• The gradient uses all available training data.
• Updates are relatively stable.
• The cost curve is easier to interpret.
• Vectorization can be applied across the complete dataset.

Disadvantages

• Each update can be slow when \(m\) is very large.
• The complete dataset must be processed before any progress is made.
• Memory requirements may be excessive.
• There is only one update per epoch.

Batch gradient descent is reasonable when the dataset is small enough to process efficiently.

The Other Extreme: Stochastic Gradient Descent

If:\[ m_{\text{batch}}=1 \]

then every example forms its own mini-batch:\[ X^{\{i\}}=x^{(i)} \]\[ Y^{\{i\}}=y^{(i)} \]

This is stochastic gradient descent, commonly abbreviated SGD.

The model updates its parameters after every example.

Advantages

• Training can begin making progress immediately.
• Parameter updates occur very frequently.
• Memory requirements per update are low.

Disadvantages

• Each gradient is an extremely noisy estimate.
• The optimization path may move in poor directions temporarily.
• The method loses most of the speed benefits of vectorization.
• Processing one example at a time is often inefficient on modern hardware.

Optimization Paths

Suppose the objective has a minimum at:\[ (W^\ast,b^\ast) \]

Batch gradient descent

Because every update uses the complete dataset, the gradient direction is relatively stable. The path may move consistently toward the minimum.

Stochastic gradient descent

Each update reflects only one example. An individual example may produce a direction that differs substantially from the complete-data gradient.

The trajectory may:

• Move toward the minimum on average
• Occasionally move in the wrong direction
• Oscillate strongly
• Continue wandering near the minimum

With a fixed learning rate, stochastic gradient descent usually does not settle exactly at the minimum.

Mini-batch gradient descent

A mini-batch gradient averages multiple examples, so it is less noisy than a single-example gradient. It is still less stable than a full-batch gradient.

Its path typically moves toward the minimum with moderate oscillation.

Why an Intermediate Size Works Well

A practical mini-batch size provides two major benefits.

Efficient vectorization

Processing many examples together allows matrix operations to take advantage of optimized numerical libraries and parallel hardware.

For example, processing 256 examples together is generally much faster than performing 256 separate single-example operations.

Frequent progress

The model updates its parameters after each mini-batch instead of waiting for the complete dataset.

If the dataset contains 5 million examples and the batch size is 1,000:\[ \frac{5{,}000{,}000}{1{,}000}=5{,}000 \]

The model performs approximately 5,000 updates per epoch.

A useful mini-batch size is large enough for efficient vectorization but small enough to allow frequent updates.

Convergence and Oscillation

Because mini-batch gradients are noisy, the model may not converge exactly to the minimum when using a fixed learning rate.

Instead, it may oscillate within a small region:\[ \theta_t \approx \theta^\ast + \text{small fluctuations} \]

This behavior can be reduced by gradually lowering the learning rate.

A smaller learning rate produces smaller updates:\[ \theta := \theta-\alpha\nabla J^{\{t\}} \]

As:\[ \alpha\rightarrow0 \]

the oscillations can become smaller, allowing the parameters to settle closer to the minimum.

Choosing the Mini-Batch Size

The best mini-batch size depends on:

• Training-set size
• Model architecture
• Size of each example
• Available CPU or GPU memory
• Hardware efficiency
• Desired update frequency
• Gradient noise
• Optimization algorithm

The mini-batch size is a hyperparameter and should be tested empirically.

Small Training Sets

If the complete training set is small, batch gradient descent is often sufficient.

A rough guideline is that if the dataset contains fewer than approximately 2,000 examples, processing the entire set at once may be reasonable.

This is not a strict boundary. The actual decision depends on the size of each example and the model’s memory requirements.

For a small dataset:\[ m_{\text{batch}}=m \]

may be the simplest and fastest option.

Typical Mini-Batch Sizes

For larger datasets, commonly tested sizes include:\[ 64,\quad128,\quad256,\quad512 \]

Larger values such as:\[ 1024 \]

are also used, although less universally.

Powers of two are common:\[ 64=2^6 \]\[ 128=2^7 \]\[ 256=2^8 \]\[ 512=2^9 \]\[ 1024=2^{10} \]

These sizes may align well with memory organization and parallel hardware, although the best choice depends on the system.

Why Powers of Two Are Common

Modern CPUs and GPUs access and process data in structured blocks. Powers of two often interact efficiently with:

• Memory alignment
• Cache organization
• GPU kernels
• Parallel processing units
• Tensor-operation libraries

This does not mean that non-power-of-two sizes are invalid. A batch size of 100 or 1,000 can still work. However, nearby powers of two are often sensible values to test.

For example, instead of:\[ 1000 \]

you might test:\[ 1024 \]

The Mini-Batch Must Fit in Memory

Every mini-batch requires memory for more than the raw inputs. Training may also store:

• Labels
• Activations
• Linear values
• Dropout masks
• Gradients
• Optimizer state
• Temporary matrix-operation buffers

A batch that barely fits based on input size alone may exceed memory once the full computation graph is included.

If the batch does not fit in CPU or GPU memory, performance can deteriorate sharply because of:

• Memory transfers
• Paging
• Repeated recomputation
• Out-of-memory errors

Choose a mini-batch size that comfortably fits the complete training computation in available memory.

Testing Candidate Sizes

A practical approach is to try several powers of two:

candidate_batch_sizes = [
    64,
    128,
    256,
    512,
]

candidate_batch_sizes = [
    64,
    128,
    256,
    512,
]

For each value, measure:

• Time per update
• Time per epoch
• Examples processed per second
• Memory usage
• Cost reduction over a fixed time
• Development performance

The fastest individual update is not necessarily the best setting. A larger batch may make each update faster per example but require more updates or different learning-rate tuning to reach the same result.

Comparison Summary

Key Takeaway

The mini-batch size determines the balance between gradient stability, update frequency, vectorization, and memory usage.

Batch gradient descent is stable but can be slow on large datasets.

Stochastic gradient descent updates frequently but is noisy and inefficiently vectorized.

Mini-batch gradient descent provides a practical balance between the two.

For small datasets, full-batch training may be appropriate. For larger datasets, start by testing powers of two such as 64, 128, 256, or 512, ensure that each batch fits comfortably in memory, and select the value that produces the fastest reliable improvement.