SynthACticBench: A Capability-Based Synthetic Benchmark for Algorithm Configuration

Margraf, Valentin ; Lappe, Anna ; Wever, Marcel ; Benjamins, Carolin ; Hüllermeier, Eyke und Lindauer, Marius (Juli 2025): SynthACticBench: A Capability-Based Synthetic Benchmark for Algorithm Configuration. Genetic and Evolutionary Computation Conference (GECCO '25), Malaga, Spain, 14. - 18. July, 2025. Proceedings of the Genetic and Evolutionary Computation Conference. ACM. S. 39-47

Abstract

Algorithm configuration deals with the automatic optimization of an algorithm's parameters to maximize its performance on a distribution of problem instances, such as Boolean satisfiability or the traveling salesperson problem. While significant progress has been made in developing optimizers for algorithm configuration - so-called algorithm configurators - their evaluation remains computationally expensive and often relies on real-world scenarios with hard-to-control characteristics. This makes it challenging to analyze their strengths and weaknesses systematically. To address this, we introduce SynthACticBench, a synthetic benchmark specifically designed to isolate and investigate key properties of algorithm configuration problems. Our benchmark distinguishes between properties related to the configuration space and those associated with the objective function. We define a configurator's ability to handle a particular property as its capability -for example, the capability to manage hierarchical configuration spaces. Using SynthACticBench, we evaluate two state-of-the-art algorithm configurators, SMAC and irace, examining their complementary capabilities and analyzing their performances across diverse benchmark functions. By providing a controlled, scalable, and capability-based evaluation environment, SynthACticBench facilitates a more targeted analysis of algorithm configurators, helping to advance research in the field. The benchmark is available at: https://github.com/annaelisalappe/SynthACticBench/.

• Dokumententyp: Konferenzbeitrag (Paper)
• EU Funded Grant Agreement Number: 101041029
• Fakultät: Mathematik, Informatik und Statistik > Informatik > Künstliche Intelligenz und Maschinelles Lernen
• Themengebiete: 000 Informatik, Informationswissenschaft, allgemeine Werke > 004 Informatik
• Sprache: Englisch