Optimization and training dynamics · MLD-0535
Hyperparameter optimization
Searching configuration choices using an evaluation procedure.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Tune learning rate and regularization.
Original caveat
Search itself can overfit validation data.
Context and aliases
Level: Intermediate. Stages: Training · Theory.
Tags: ML · DL · SLM · LLM.
Aliases: HPO.
Related concepts
Concepts linking here
Technology connections
- BoTorch — Bayesian optimization toolkit built on PyTorch.
- ClearML — Experiment tracking and ML workflow platform.
- DVC — Data and pipeline versioning linked to Git workflows.
- lakeFS — Version-control-like operations for object storage.
- MLflow — Experiment, artifact, model and AI lifecycle tooling.
- Optuna — Hyperparameter optimization library.
- Ray Tune — Distributed experiment and hyperparameter tuning library.
- TensorBoard — Visualization tools for model training logs and artifacts.
- Weights & Biases — Experiment tracking and model-development platform.
Original references
- Deep Learning: online textbook ↗ · Consulted
- PyTorch optimization algorithms ↗ · Reference link
- PyTorch documentation ↗ · Reference link
- Original reference URL ↗