Relatively Rational: Learning Utilities and Rationalities Jointly from Pairwise Preferences

Yamagata, Taku; Oberkofler, Tobias Florin; Kaufmann, Timo ; Bengs, Viktor ; Hüllermeier, Eyke und Santos-Rodriguez, Raul (26. July 2024): Relatively Rational: Learning Utilities and Rationalities Jointly from Pairwise Preferences. ICML 2024 Workshop on Models of Human Feedback for AI Alignment, Vienna, Austria, 26. July 2024. Vienna, Austria:

Abstract

Learning utilities from preference feedback has become increasingly important, particularly in fine-tuning language models such as ChatGPT. Traditional methods often assume equal rationality among labellers, leading to inaccurate utility estimates. We propose an algorithm that jointly estimates trainer rationality and item utilities to enhance utility learning and gain additional insights from feedback. Our approach focuses on settings where feedback is received from multiple trainers, using the Boltzmann-rational model to relate choices to latent utilities while accounting for varying levels of rationality. Given shared utilities, our method identifies rationality ratios among trainers from observed choices without extra calibration data or assumptions. We analyse the theoretical impact of assuming equal rationality on utility accuracy and empirically show superior performance in an action-advice setting, where agents construct policies using the learned utilities as rewards. By accurately modelling trainer rationality, we can enhance high-quality feedback collection, potentially leading to better-aligned models and an improved understanding of human preferences.

• Item Type: Conference or Workshop Item (Paper)
• EU Funded Grant Agreement Number: 101073307
• Faculties: Mathematics, Computer Science and Statistics > Computer Science > Artificial Intelligence and Machine Learning
• Subjects: 000 Computer science, information and general works > 004 Data processing computer science
• Place of Publication: Vienna, Austria