A Hyperparameter Tuning Strategy That Fits Your Budget

A hyperparameter is a choice made outside the parameter-fitting updates, such as learning rate, model width, or dropout probability. One trial trains and evaluates one configuration of these choices. Tuning searches for a configuration that performs well within a limited budget, so the evaluation protocol and the cost of each trial matter alongside the search algorithm.

Choose what to search from the baseline’s behavior

Start from a working baseline and identify what currently limits it: unstable optimization, overfitting, insufficient capacity, or training cost. The useful search order depends on that diagnosis. The following choices address different problems; their positions are not a universal ranking.

For an established architecture, learning rate is often a useful first search. Once training progresses reliably, use validation behavior to choose whether to spend the remaining budget on regularization, capacity, or another constraint.

How to decide which improvement to attempt next is the subject of A Systematic Strategy for Improving Machine Learning Models.

Define what a trial costs and what it measures

Choose a validation metric aligned with the eventual decision, and state whether larger or smaller is better. Keep the data split fixed during a search, fit preprocessing on training data only, and reserve the test set for evaluation after selection. A trial configuration includes the training duration and schedule, not just optimizer settings.

Measure a pilot run on the intended hardware. If one full run takes two GPU-hours and the total allowance is 24 GPU-hours, at most 12 such runs fit before overhead. Reserving three full runs to check the selected configuration across seeds leaves 18 GPU-hours for search. Wider models may take longer, so counting trials alone can hide a budget overrun. Record actual elapsed time and leave room for validation, checkpointing, and failures. Parallel workers reduce elapsed search time but still consume the sum of their compute time.

Equal epochs compare equal passes over the data; equal update counts compare optimizer steps; equal GPU-hours compare performance under a compute limit. These answer different questions, especially when batch size or architecture changes. Choose the resource before ranking trials. Stop starting new trials when the remaining budget cannot cover them and the reserved confirmation runs.

When random search covers more useful values

With a \(5\times5\) grid over two parameters and 25 trials, each coordinate takes only five values. If only one parameter matters, the other dimension repeats those five settings. Independent random draws from continuous distributions give 25 distinct values of each coordinate with probability one in ideal arithmetic. Discrete choices, rounding, and finite precision can produce repeats.

When only a few coordinates strongly influence performance, this coverage can make random search more effective than an evenly allocated grid. It does not guarantee a better winner. A small grid remains useful for a few meaningful discrete choices or a controlled comparison. Either search can be extended with more points; random sampling makes that extension easy without requiring a rectangular grid.

Sample on the scale that matches the parameter

Uniform sampling from \([0.0001,0.1]\) places about 90.1% of its probability above \(0.01\), and about 0.9% between \(10^{-4}\) and \(10^{-3}\). This is appropriate only if equal absolute intervals deserve equal search effort. When the plausible scale spans orders of magnitude, sample the exponent uniformly: each tenfold interval then receives equal probability. The useful range depends on the optimizer, model, and training setup.

import numpy as np

rng = np.random.default_rng(0)

def log_uniform(low, high, size, rng):
    if not (np.isfinite(low) and np.isfinite(high) and 0 < low < high):
        raise ValueError("bounds must be finite and satisfy 0 < low < high")
    return 10 ** rng.uniform(np.log10(low), np.log10(high), size)

lr = log_uniform(1e-5, 1e-1, 8, rng)
# Sampling the distance from 1 varies the momentum memory scale.
beta = 1 - log_uniform(1e-3, 0.3, 5, rng)
print(np.sort(np.round(lr, 6)))
print(np.sort(np.round(beta, 4)))
# [1.2000e-05 1.5000e-05 1.2000e-04 2.6700e-03 3.5310e-03 8.2790e-03
#  1.7909e-02 4.4774e-02]
# [0.7928 0.867  0.8951 0.9778 0.999 ]

Positive learning rates and regularization coefficients are often searched logarithmically when several orders of magnitude are plausible. Zero cannot be included in a log-uniform interval: test “no weight decay” as a separate choice. Widths and head counts need valid discrete values, including divisibility constraints; rounding arbitrary continuous samples can distort their probabilities. Linear sampling over a chosen range can be a starting point for dropout or label smoothing, but it is not dictated by an additive effect on the model.

For an exponential moving average with decay \(\beta\), the memory scale is roughly \(1/(1-\beta)\) updates. Thus \(\beta=0.99\) and \(0.999\) correspond to roughly 100 and 1,000 updates. Sampling \(1-\beta\) on a log scale explores this range more evenly. Values 0.5 and 0.6 instead give scales 2 and 2.5; neither comparison establishes how sensitive a particular optimizer will be.

Coarse to fine

Start with plausible broad ranges and use pilot trials to discover failures and promising regions. Extend a few configurations to the intended training duration before narrowing the search. If strong results sit at a range boundary, consider extending that boundary; if several regions look competitive, retain more than one. The number of rounds should follow the remaining budget and evidence, not a fixed two- or three-round rule.

Short trials can rank configurations differently from full trials. Learning rate, regularization, warmup, and schedule duration can all affect how quickly progress appears. A trial stopped inside warmup says little about its eventual performance. Use pilot learning curves to judge whether early scores are informative, and reserve some full-duration comparisons for settings that might improve later. Early elimination saves compute by accepting a risk of discarding the eventual best configuration.

Successive halving

Successive halving evaluates many configurations at a small resource budget, keeps a fraction, and increases the resource for the survivors. Here \(b\) is the initial budget per configuration and the integer reduction factor \(\eta>1\) controls both promotion and budget growth. With 27 configurations, \(b=1\), \(\eta=3\), and three rounds, the stages evaluate 27 at budget 1, 9 at budget 3, and 3 at budget 9. The winner is selected from the final three.

def successive_halving(configs, evaluate, min_budget=1, eta=3, rounds=3):
    """evaluate(config, total_budget) returns a finite score; higher is better.

    The evaluator owns training and any checkpoint reuse.
    """
    survivors = list(configs)
    if not survivors:
        raise ValueError("at least one configuration is required")
    if not isinstance(eta, int) or eta < 2:
        raise ValueError("eta must be an integer >= 2")
    if not isinstance(rounds, int) or rounds < 1:
        raise ValueError("rounds must be a positive integer")
    if not isinstance(min_budget, int) or min_budget < 1:
        raise ValueError("min_budget must be a positive integer")
    budget = min_budget
    for stage in range(rounds):
        scored = [(float(evaluate(c, budget)), c) for c in survivors]
        if not all(np.isfinite(score) for score, _ in scored):
            raise ValueError("evaluation scores must be finite")
        scored.sort(key=lambda item: item[0], reverse=True)
        if stage == rounds - 1:
            return scored[0][1]
        keep = max(1, len(scored) // eta)
        survivors = ]
        budget *= eta

calls = []
def toy_evaluate(config, total_budget):
    calls.append((config, total_budget))
    return 1 - (config - 13) ** 2 / 400 - 1 / total_budget

winner = successive_halving(range(27), toy_evaluate)
for budget in (1, 3, 9):
    print("budget", budget, "trials", sum(b == budget for _, b in calls))
print("winner", winner)
print("restart resource", sum(b for _, b in calls))
previous = {}
resumed_resource = 0
for config, budget in calls:
    resumed_resource += budget - previous.get(config, 0)
    previous[config] = budget
print("resumed resource", resumed_resource)
# budget 1 trials 27
# budget 3 trials 9
# budget 9 trials 3
# winner 13
# restart resource 81
# resumed resource 63

The toy score keeps the same ranking at every budget so that the example isolates allocation; it does not establish that early rankings predict real training outcomes. If each stage restarts training, the cost is \(27\times1+9\times3+3\times9=81\) units, the cost of nine full budget-9 runs. If survivors resume checkpoints, the additional cost is \(27\times1+9\times2+3\times6=63\) units, equivalent to seven full runs. These counts ignore overhead and assume equal cost per resource unit across configurations.

In a training evaluator, total_budget=3 means train to a cumulative resource of 3, not add 3 to the previous 1. Resuming must restore model, optimizer, scheduler, and random-generator state. If the resource is updates, keep the intended full-run schedule fixed and truncate it at each stage; compressing an entire cosine schedule into each short trial tests a different recipe. The example fails visibly on a nonfinite score; a production search should record failed trials and apply a declared failure policy.

Hyperband runs several halving brackets with different starting counts and initial budgets. Some brackets spend more per candidate before pruning, reducing dependence on very early scores. This offers protection against slow starters, not a guarantee that the best configuration survives. The usefulness of early performance must be assessed for the training recipe, including its learning rate and regularization.

When Bayesian optimization is worth it

Bayesian optimization fits a surrogate, a model predicting the score and uncertainty of untried configurations. An acquisition rule uses those predictions to choose what to evaluate next. It can favor a region with a strong predicted score or an uncertain region that might improve on it. After the trial, the observed result updates the surrogate. This is useful when evaluations are expensive enough that choosing the next point carefully can justify the modeling overhead.

There is no universal cutoff at 10 trials, 20 dimensions, or 100 runs. Noise, prior information, discrete choices, and how many trials can run simultaneously affect the benefit. Batch and asynchronous Bayesian methods can use parallel workers too. Random search is a useful baseline when simplicity and low coordination cost matter; compare methods using the same search space and total evaluation budget.

Interactions that can change the preferred setting

A sweep with other settings fixed answers a conditional question: what works best under those fixed settings? It is a valid comparison, but its winner may change when the surrounding configuration changes. Jointly search a small number of interacting choices when the budget permits.

• Learning rate and batch size: changing batch size changes update noise and, at fixed epochs, the number of updates. A larger batch may support a higher learning rate, but stability is not guaranteed. Declare the training budget when comparing them.
• Learning rate and warmup: a higher peak may benefit from more warmup, but extending warmup does not guarantee a stable peak rate.
• Model capacity and regularization: a width sweep at fixed dropout measures that combination. A wider model does not necessarily need stronger regularization; compare a few strengths if the validation curves justify it.
• Weight decay and learning rate: AdamW separates decay from the adaptive gradient calculation, but its shrinkage still includes \(1-\alpha_t\lambda\). For example, with zero data gradient and zero moment state, \(w=2\), \(\alpha=0.1\), and \(\lambda=0.01\) give \(w_{\text{new}}=1.998\). Doubling \(\alpha\) gives 1.996. Matching the product does not match the full training dynamics because learning rate also scales the data-gradient update.
• Augmentation strength and duration: stronger augmentation changes the training distribution and may change how long learning takes. Training losses under different augmentation settings are not directly interchangeable; compare the common validation objective.

How to find a learning rate and shape it over training is covered in Learning Rate Schedules and Warmup in Deep Learning.

Keep selection separate from final evaluation

Select configurations on validation data, then evaluate the frozen choice on an untouched test set. Repeatedly maximizing a noisy validation score can select favorable noise as well as better models. The size of that selection effect depends on score uncertainty, candidate dependence, and the search procedure; candidate count alone does not imply a one- or two-point gap. If test results drive another tuning decision, that set has become part of selection and fresh independent evaluation is needed.

Record every trial’s complete configuration, data split, code version, seed, resource use, and learning curves, including failures. Re-run the leading configurations and the baseline under a shared set of seeds before trusting a small difference. Report the individual scores and their variation; two or three seeds offer an initial check, not a reliable estimate of every small effect. Repeated seeds on a fixed split examine training randomness, not uncertainty from sampling a new dataset. Reserve enough compute for these comparisons before the search consumes it.

Seeding, configuration, and checkpointing for runs like this are covered in Reproducible Training Pipelines: Seeds, Configs, and Checkpoints.

Exercises

1. Log scale, demonstrated. Sample 1,000 learning rates uniformly from \([10^{-5},10^{-1}]\) and 1,000 log-uniformly from the same range. For each, report the fraction that land below \(10^{-3}\).

You should get: a tiny fraction from uniform sampling and roughly half from log-uniform.

import numpy as np
rng = np.random.default_rng(0)
lin = rng.uniform(1e-5, 1e-1, 1000)
log = 10 ** rng.uniform(np.log10(1e-5), np.log10(1e-1), 1000)
print("uniform     ", round(float((lin < 1e-3).mean()), 4))
print("log-uniform ", round(float((log < 1e-3).mean()), 4))
# uniform      0.01
# log-uniform  0.534

2. Coordinate coverage. Compare a \(5\times5\) grid with 25 independent random trials over two continuous parameters, where only one affects the score. How many distinct values of that parameter does each test in ideal arithmetic? Generalize to an equal-resolution grid with \(k\) values on each of \(d\) axes, and \(n=k^d\) trials.

Distinguish the number of distinct coordinates from the probability of finding a useful region.

3. Selection overfits too. Simulate 500 candidates with true accuracy 80%. Give each an independent binomial validation score from 1,000 trials. Report the best observed score and its gap to 80%. How does this idealization differ from models evaluated on the same validation examples?

You should get: a best score noticeably above 80%, achieved by a model that is not actually better.

import numpy as np
rng = np.random.default_rng(0)
scores = rng.binomial(1000, 0.80, size=500) / 1000
print("best", scores.max(), "| mean", round(float(scores.mean()), 4),
      "| gap", round(float(scores.max() - 0.80), 4))
# best 0.842 | mean 0.7996 | gap 0.042

References

• Bergstra and Bengio (2012). Random Search for Hyper-Parameter Optimization. JMLR.
• Snoek, Larochelle, and Adams (2012). Practical Bayesian Optimization of Machine Learning Algorithms. NIPS. See also BoTorch’s batch acquisition documentation for parallel candidate selection.
• Jamieson and Talwalkar (2016). Non-stochastic Best Arm Identification and Hyperparameter Optimization. AISTATS.
• Li, Jamieson, DeSalvo, Rostamizadeh, and Talwalkar (2018). Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. JMLR.
• Loshchilov and Hutter. Decoupled Weight Decay Regularization.