Symbolic Explanations for Hyperparameter Optimization

Segel, Sarah ; Graf, Helena ; Tornede, Alexander ; Bischl, Bernd und Lindauer, Marius (2023): Symbolic Explanations for Hyperparameter Optimization. International Conference on Automated Machine Learning, Potsdam, Germany, 12-15 November 2023. Faust, Aleksandra; Garnett, Roman; White, Colin; Hutter, Frank und Gardner, Jacob R. (Hrsg.): In: Proceedings of Machine Learning Research, Bd. 224 Cambridge, MA: JMLR.

Abstract

Hyperparameter optimization (HPO) methods can determine well-performing hyperparameter configurations efficiently but often lack insights and transparency. We propose to apply symbolic regression to meta-data collected with Bayesian optimization (BO) during HPO. In contrast to prior approaches explaining the effects of hyperparameters on model performance, symbolic regression allows for obtaining explicit formulas quantifying the relation between hyperparameter values and model performance. Overall, our approach aims to make the HPO process more explainable and human-centered, addressing the needs of multiple user groups: First, providing insights into the HPO process can support data scientists and machine learning practitioners in their decisions when using and interacting with HPO tools. Second, obtaining explicit formulas and inspecting their properties could help researchers understand the HPO loss landscape better. In an experimental evaluation, we find that naively applying symbolic regression directly to meta-data collected during HPO is affected by the sampling bias introduced by BO. However, the true underlying loss landscape can be approximated by fitting the symbolic regression on the surrogate model trained during BO. By penalizing longer formulas, symbolic regression furthermore allows the user to decide how to balance the accuracy and explainability of the resulting formulas.

• Dokumententyp: Konferenzbeitrag (Paper)
• Fakultät: Mathematik, Informatik und Statistik > Statistik
• Themengebiete: 000 Informatik, Informationswissenschaft, allgemeine Werke > 004 Informatik ; 300 Sozialwissenschaften > 310 Statistiken
• Ort: Cambridge, MA
• Bemerkung: Artikel No : 2
• Sprache: Englisch