Best Arm Identification with Retroactively Increased Sampling Budget for More Resource-Efficient HPO

Brandt, Jasmin; Wever, Marcel ; Bengs, Viktor und Hüllermeier, Eyke (August 2024): Best Arm Identification with Retroactively Increased Sampling Budget for More Resource-Efficient HPO. Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI-24), Jeju, South Korea, 3. - 9. August 2024. Larson, Kate (Hrsg.): In: Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence, International Joint Conferences on Artificial Intelligence Organization. S. 3742-3750

Abstract

Hyperparameter optimization (HPO) is indispensable for achieving optimal performance in machine learning tasks. A popular class of methods in this regard is based on Successive Halving (SHA), which casts HPO into a pure-exploration multi-armed bandit problem under finite sampling budget constraints. This is accomplished by considering hyperparameter configurations as arms and rewards as the negative validation losses. While enjoying theoretical guarantees as well as working well in practice, SHA comes, however, with several hyperparameters itself, one of which is the maximum budget that can be allocated to evaluate a single arm (hyperparameter configuration). Although there are already solutions to this meta hyperparameter optimization problem, such as the doubling trick or asynchronous extensions of SHA, these are either practically inefficient or lack theoretical guarantees. In this paper, we propose incremental SHA (iSHA), a synchronous extension of SHA, allowing to increase the maximum budget a posteriori while still enjoying theoretical guarantees. Our empirical analysis of HPO problems corroborates our theoretical findings and shows that iSHA is more resource-efficient than existing SHA-based approaches.

• Dokumententyp: Konferenzbeitrag (Paper)
• Fakultät: Mathematik, Informatik und Statistik > Informatik > Künstliche Intelligenz und Maschinelles Lernen
• Themengebiete: 000 Informatik, Informationswissenschaft, allgemeine Werke > 004 Informatik
• Sprache: Englisch