Hyperparameters

Hyperparameters are configuration settings that control the learning process but are not optimized directly from training data—instead, they must be set before training begins. Unlike model parameters (weights and biases in neural networks) which algorithms adjust to minimize loss, hyperparameters remain fixed throughout training and fundamentally shape how algorithms operate. Hyperparameter choices dramatically influence whether models succeed or fail, affecting training efficiency, final model performance, generalization ability, and convergence properties.​

Hyperparameter types vary across algorithms but include:

Optimization Hyperparameters: Learning rate controls step size in gradient-based optimization—too high causes divergence, too low causes slow convergence. Batch size determines how many training examples are processed before updating parameters; larger batches are more stable but computationally expensive. Number of epochs specifies how many complete passes through the training data to perform.​

Model Architecture Hyperparameters: For neural networks, number of layers, layer width(neurons per layer), and activation functions determine network capacity and expressiveness. Kernel functions in support vector machines and tree depth in decision trees control model complexity.​

Regularization Hyperparameters: Regularization strength (L1/L2 penalty weights) controls the tradeoff between fitting training data and maintaining model simplicity. Dropout rate in neural networks controls the fraction of neurons randomly deactivated during training.​

Data Hyperparameters: Train-test split ratio, cross-validation fold count, and data augmentation intensity control data partitioning and preprocessing.​

Grid Search: Exhaustively evaluates all combinations of hyperparameter values from predefined sets. While straightforward, grid search becomes computationally prohibitive as the number of hyperparameters or value options increases (curse of dimensionality).​

Random Search: Samples random combinations of hyperparameter values, often outperforming grid search because some hyperparameters matter more than others. Random search efficiently allocates budget toward the most important hyperparameters.​

Bayesian Optimization: Uses a probabilistic surrogate model (typically Gaussian Process regression) to model the relationship between hyperparameter configurations and performance. An acquisition function (such as Expected Improvement) balances exploration (trying new regions) against exploitation (focusing on promising regions discovered so far). Bayesian optimization efficiently finds good hyperparameters with fewer evaluations than grid or random search, particularly valuable when model training is expensive.​

Hyperband: Combines successive halving with random search, iteratively evaluating many configurations with small resource budgets, eliminating poorly performing configurations quickly, and concentrating resources on promising ones. Hyperband achieves superior performance per unit of computational cost compared to Bayesian optimization and traditional methods.​

Multi-Fidelity Optimization: Leverages cheaper approximations (smaller datasets, fewer iterations) alongside expensive full evaluations to guide search efficiently.​

Parallel Evaluation: Modern methods enable evaluating multiple hyperparameter configurations simultaneously, essential for large-scale hyperparameter optimization.​

Automated Machine Learning (AutoML): Systems like Amazon SageMaker Automatic Model Tuning automate hyperparameter selection completely, discovering dataset-specific defaults more quickly than manual tuning.​

Meta-Learning: Learning good hyperparameter settings from prior tasks enables faster configuration for new datasets, particularly valuable in settings with many similar problems.​

Computational Cost: Hyperparameter optimization requires evaluating many configurations, each involving training a complete model—computationally expensive for large-scale datasets or complex architectures.​

High-Dimensional Search Spaces: As algorithm complexity increases, the number of hyperparameters grows, making exhaustive search infeasible.​

Dataset Dependence: Optimal hyperparameter values vary significantly across datasets; settings performing excellently on one dataset may fail on another.​

Preprocessing Hyperparameters Often Overlooked: While attention focuses on algorithm hyperparameters, preprocessing hyperparameters (feature scaling, normalization, missing value imputation) profoundly impact performance but remain frequently untuned.​

Documentation and Reproducibility: Proper hyperparameter documentation is essential but often neglected, undermining model reproducibility and robustness checks.