Research / IDEAL 2023 (LNCS)

Rapid and Low-Cost Evaluation of Multi-fidelity Scheduling Algorithms for Hyperparameter Optimization

Aakash Mishra, Jeremy Hsu, Frank D'Agostino, Utku Sirin, Stratos Idreos

IDEAL 2023 (LNCS) · Nov 2023

(a) Original CIFAR-10 ConvNet tuning results next to (b) PHOSS's simulated cumulative regret for random search, ASHA, SHA and PBT

PHOSS simulates training-loss landscapes so multi-fidelity hyperparameter schedulers can be compared quickly and cheaply, without training real models.

Overview

Comparing multi-fidelity schedulers for hyperparameter optimization normally means training many real models, which is slow and expensive. We introduce PHOSS, an open-source Python platform that generates synthetic loss landscapes from a chosen distribution and workload, runs distributed hyperparameter searches over them with Ray Tune, and reports regret and loss curves across seeds. This makes it fast and low-cost to compare schedulers across many configurations.

Cite

@inproceedings{mishra2023rapid,
  title = {Rapid and Low-Cost Evaluation of Multi-fidelity Scheduling Algorithms for Hyperparameter Optimization},
  author = {Aakash Mishra and Jeremy Hsu and Frank D'Agostino and Utku Sirin and Stratos Idreos},
  year = {2023},
  booktitle = {Intelligent Data Engineering and Automated Learning (IDEAL 2023)},
  pages = {64--69},
  publisher = {Springer},
  doi = {10.1007/978-3-031-48232-8_7}
}