Improving Labeling Through Social Science Insights: Results and Research Agenda

Beck, Jacob ; Eckman, Stephanie ; Chew, Rob and Kreuter, Frauke (2022): Improving Labeling Through Social Science Insights: Results and Research Agenda. 24th International Conference on Human-Computer Interaction, HCII 2022, Virtual Event, June 26 – July 1, 2022. In: HCI International 2022 – Late Breaking Papers: Interacting with eXtended Reality and Artificial Intelligence. 24th International Conference on Human-Computer Interaction, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings, Lecture Notes in Computer Science Vol. 13518 Cham, Switzerland: Springer. pp. 245-261

Abstract

Frequently, Machine Learning (ML) algorithms are trained on human-labeled data. Although often seen as a “gold standard,” human labeling is all but error free. Decisions in the design of labeling tasks can lead to distortions of the resulting labeled data and impact predictions. Building on insights from survey methodology, a field that studies the impact of instrument design on survey data and estimates, we examine how the structure of a hate speech labeling task affects which labels are assigned. We also examine what effect task ordering has on the perception of hate speech and what role background characteristics of annotators have on classifications provided by annotators. The study demonstrates the importance of applying design thinking at the earliest steps of ML product development. Design principles such as quick prototyping and critically assessing user interfaces are not only important in interaction with end users of an artificial intelligence (AI)-driven products, but are crucial early in development, prior to training AI algorithms.

• Item Type: Conference or Workshop Item (Paper)
• Faculties: Mathematics, Computer Science and Statistics > Statistics
• Subjects: 300 Social sciences > 310 Statistics
• ISSN: 0302-9743
• Place of Publication: Cham, Switzerland
• Language: English