The (black) art of runtime evaluation: Are we comparing algorithms or implementations?

Kriegel, Hans-Peter; Schubert, Erich und Zimek, Arthur (2017): The (black) art of runtime evaluation: Are we comparing algorithms or implementations? In: Knowledge and information Systems, Bd. 52, Nr. 2: S. 341-378

Abstract

Any paper proposing a new algorithm should come with an evaluation of efficiency and scalability (particularly when we are designing methods for "big data"). However, there are several (more or less serious) pitfalls in such evaluations. We would like to point the attention of the community to these pitfalls. We substantiate our points with extensive experiments, using clustering and outlier detection methods with and without index acceleration. We discuss what we can learn from evaluations, whether experiments are properly designed, and what kind of conclusions we should avoid. We close with some general recommendations but maintain that the design of fair and conclusive experiments will always remain a challenge for researchers and an integral part of the scientific endeavor.

• Dokumententyp: Zeitschriftenartikel
• Fakultät: Mathematik, Informatik und Statistik > Informatik
• Themengebiete: 000 Informatik, Informationswissenschaft, allgemeine Werke > 004 Informatik
• ISSN: 0219-1377
• Sprache: Englisch