Systematic Hyperparameter Tuning for Neural Networks
Training a neural network involves choosing many hyperparameters. Because testing every possible combination is usually impractical, the goal is to organize the search so that computational resources are spent on the parameters that matter most.
Two particularly useful principles are:
Sample hyperparameter combinations randomly instead of using a fixed grid.
Begin with a broad search, identify promising regions, and then search those regions more precisely.
Common Neural Network Hyperparameters
A neural network may require decisions about:
- Learning rate \(\alpha\)
- Momentum coefficient \(\beta\)
- Adam parameters \(\beta_1\), \(\beta_2\), and \(\epsilon\)
- Number of hidden layers
- Number of hidden units in each layer
- Mini-batch size
- Learning rate decay
- Regularization strength
- Activation functions
These hyperparameters do not have equal influence on performance. Some deserve much more attention than others.
Which Hyperparameters Matter Most?
The exact priority depends on the model and dataset, but the following hierarchy is a useful starting point.
First priority: learning rate
The learning rate \(\alpha\) is often the most important hyperparameter.
It determines the size of each optimization step:\[ W^{[l]} \leftarrow W^{[l]}-\alpha\,dW^{[l]} \]
If the learning rate is too small, training can be extremely slow. If it is too large, optimization may oscillate or diverge.
A well-chosen learning rate can make the difference between a model that trains effectively and one that makes almost no useful progress.
Second priority
After the learning rate, useful parameters to investigate include:
- Momentum coefficient
- Mini-batch size
- Number of hidden units
For momentum, the following is a strong default:\[ \beta=0.9 \]
The mini-batch size influences both computational efficiency and gradient noise. The number of hidden units affects the model’s capacity to learn complex patterns.
Third priority
The following can also have a substantial effect:
- Number of hidden layers
- Learning rate decay schedule
The number of layers may dramatically affect performance when depth is important for the problem. Learning rate decay can improve convergence by reducing the step size later in training.
Adam parameters
Adam commonly uses:\[ \beta_1=0.9 \]\[ \beta_2=0.999 \]\[ \epsilon=10^{-8} \]
These values work well across many applications and are usually left at their defaults. The learning rate used with Adam remains important and should generally be tuned.
A Practical Priority Table
| Priority | Hyperparameters | General guidance |
|---|---|---|
| Highest | Learning rate \(\alpha\) | Almost always tune |
| High | Mini-batch size, momentum coefficient, hidden units | Tune when resources permit |
| Moderate | Number of layers, learning rate decay | Explore after the basic optimization setup works |
| Usually low | Adam’s \(\beta_1\), \(\beta_2\), and \(\epsilon\) | Begin with standard defaults |
This ranking is a guideline rather than a universal law. The most important hyperparameters can vary across architectures and applications.
Grid Search
Suppose a model has two hyperparameters, \(h_1\) and \(h_2\). A traditional grid search selects several values for each one and evaluates every combination.
For example, selecting five values for each parameter creates:\[ 5 \times 5 = 25 \]
experiments.
A grid might look conceptually like this:\[ \begin{aligned} h_1 &\in \{h_{1,1},h_{1,2},h_{1,3},h_{1,4},h_{1,5}\}\\ h_2 &\in \{h_{2,1},h_{2,2},h_{2,3},h_{2,4},h_{2,5}\} \end{aligned} \]
Grid search can work when:
- There are very few hyperparameters.
- Every parameter has roughly equal importance.
- The search space is small.
- Individual experiments are inexpensive.
However, neural-network hyperparameters rarely have equal importance.
Why Grid Search Can Be Inefficient
Suppose:
- \(h_1\) is the learning rate \(\alpha\)
- \(h_2\) is Adam’s numerical-stability parameter \(\epsilon\)
The learning rate may strongly affect performance, while changing \(\epsilon\) may have almost no visible effect.
A \(5 \times 5\) grid evaluates 25 combinations, but it tests only five distinct learning rates. Most of the experiments simply repeat those learning rates with different values of a relatively unimportant parameter.
This becomes increasingly wasteful as the number of hyperparameters grows.
If five values are tested for each of three parameters, the search requires:\[ 5^3=125 \]
experiments.
For six parameters, it requires:\[ 5^6=15{,}625 \]
The number of combinations grows exponentially with the number of hyperparameters.
Random Search
Random search independently samples hyperparameter combinations from specified ranges or distributions.
With 25 random experiments involving two continuous hyperparameters, the process can evaluate 25 different values of each parameter rather than only five distinct values per parameter.
If one parameter turns out to be much more important than the other, random search provides much broader coverage of that important dimension.
Conceptual example
import numpy as np
experiments = []
for _ in range(25):
learning_rate = 10 ** np.random.uniform(-5, -1)
beta = np.random.uniform(0.8, 0.999)
experiments.append({
"learning_rate": learning_rate,
"beta": beta,
})This example samples 25 distinct combinations of the learning rate and momentum coefficient.
Why Random Search Works Better
The main advantage of random search is not randomness by itself. Its advantage is that it explores more distinct values along every dimension.
Suppose only one of ten hyperparameters has a major effect on performance. A grid may repeatedly test the same few values of that important parameter while varying nine relatively unimportant parameters.
Random search is more likely to explore many values of the important parameter—even when its importance was not known beforehand.
Random search protects against uncertainty about which hyperparameters matter most.
This is especially valuable in high-dimensional search spaces, where visualizing or manually designing a comprehensive grid becomes difficult.
Sampling in Higher Dimensions
With two hyperparameters, the search space can be visualized as a square. With three, it becomes a cube. With more than three, it becomes a higher-dimensional region.
Each random experiment selects one point:\[ (h_1,h_2,\ldots,h_k) \]
from the search space.
This allows every experiment to explore a new value along almost every continuous dimension. The same principle therefore becomes even more useful as the number of hyperparameters increases.
Coarse-to-Fine Search
Random search can be combined with a coarse-to-fine strategy.
Stage 1: broad exploration
Begin by sampling across a wide range of possible values. The objective is to identify promising regions rather than immediately locate the exact best configuration.
For example:\[ \alpha \in [10^{-5},10^{-1}] \]
The initial search explores this entire range.
Stage 2: identify a promising region
After evaluating the initial configurations, inspect the best-performing results.
Suppose the most successful learning rates are concentrated around:\[ \alpha \in [10^{-3},10^{-2}] \]
This suggests that the narrower interval deserves more attention.
Stage 3: refine the search
Run another random search with denser sampling inside the promising region:\[ \alpha \in [10^{-3},10^{-2}] \]
The same refinement can be applied simultaneously to several hyperparameters.
Stage 4: repeat when worthwhile
If the second search reveals an even smaller promising region, refine it again. Continue only while the expected benefit justifies the computational cost.
Evaluating Hyperparameter Configurations
Each configuration should be evaluated according to a clearly defined objective.
Possible evaluation criteria include:
- Development-set loss
- Development-set accuracy
- Precision, recall, or F1 score
- Training speed
- Memory consumption
- Inference latency
- A domain-specific business or scientific metric
Training-set performance alone is usually not sufficient for model selection because a configuration may fit the training data well while generalizing poorly.
The development set is generally the appropriate place to compare configurations. The test set should remain separate from routine tuning so that it can provide a less biased final evaluation.
A Practical Search Procedure
A useful hyperparameter-tuning workflow is:
- Choose a metric for comparing configurations.
- Identify the hyperparameters likely to have the greatest effect.
- Define broad but reasonable ranges.
- Randomly sample configurations.
- Train each model under comparable conditions.
- Evaluate every configuration using the same development metric.
- Identify promising regions.
- Search those regions more densely.
- Select the strongest configuration.
- Confirm its performance with a sufficiently complete training run.
Keep Experiment Records
Systematic tuning depends on accurate experiment tracking. For every run, record at least:
- Hyperparameter values
- Model architecture
- Random seed
- Training and development metrics
- Number of epochs or update steps
- Training duration
- Hardware configuration
- Notes about instability, divergence, or resource limitations
A simple record might look like this:
result = {
"learning_rate": 0.001,
"mini_batch_size": 128,
"beta": 0.9,
"hidden_units": [256, 128, 64],
"development_loss": 0.184,
"development_accuracy": 0.937,
"training_time_minutes": 42.6,
}Without reliable records, it is easy to repeat experiments, compare incompatible runs, or lose a promising configuration.
Key Takeaway
Start by tuning the learning rate, use sensible defaults for relatively stable parameters, and prefer random search over a rigid grid. After locating a promising region, apply a coarse-to-fine search to explore it more precisely.