Hyperparameter

Definition

A hyperparameter is a setting in a machine learning algorithm that is not learned from the data during training, but instead set before training begins.

• They control how the model learns or how it is structured.
• Contrast with parameters (weights, biases) → learned automatically by the model.

---

Examples of Hyperparameters

1. Model Structure
   • Number of layers in a neural network.
   • Number of neurons per layer.
   • Maximum depth of a decision tree.
2. Training Process
   • Learning rate (step size for weight updates).
   • Batch size.
   • Number of epochs.
   • Dropout rate (in NN).
3. Regularization
   • L1/L2 penalty strength (Ridge, Lasso).
   • Weight decay.
4. Optimization Algorithms
   • Momentum parameter (for SGD).
   • Beta values in Adam optimizer.

---

Why Hyperparameters Matter

• Different hyperparameter choices can lead to very different performance.
• Bad choices → underfitting, overfitting, slow training.
• Good tuning = critical for maximizing accuracy and generalization.

---

Hyperparameters vs Parameters

Parameters (learned)	Hyperparameters (set manually/externally)
Learned during training (via optimization)	Defined before training
Example: weights, biases	Example: learning rate, # of layers
Represent model’s “knowledge”	Represent model’s “learning strategy”

---

Tuning Hyperparameters

1. Manual tuning → try different values based on intuition.
2. Grid search → test across a full set of combinations.
3. Random search → randomly sample combinations (often more efficient).
4. Bayesian optimization → use probability models to search smartly.
5. Hyperband / Population-based tuning → efficient large-scale methods.

---

Example

• Train a neural net on MNIST digits.
• Hyperparameters:
  • Learning rate = 0.001
  • Batch size = 64
  • Dropout rate = 0.5
  • Epochs = 20
• Model learns parameters (weights) automatically, but performance depends heavily on chosen hyperparameters.

---

Summary
Hyperparameters = configuration choices set before training that control how a model learns or is structured.

• Examples: learning rate, tree depth, dropout rate.
• Must be tuned carefully (grid search, random search, Bayesian).