ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning

Kaufmann, Timo ; Metz, Yannick; Keim, Daniel A. und Hüllermeier, Eyke (December 2025): ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning. 39th Conference on Neural Information Processing Systems (NeurIPS 2025), San Diego, CA, USA, 2.- 7. December 2025. Advances in Neural Information Processing Systems. Vol. 38 San Diego, CA, USA:

Abstract

Binary choices, as often used for reinforcement learning from human feedback (RLHF), convey only the direction of a preference. A person may choose apples over oranges and bananas over grapes, but which preference is stronger? Strength is crucial for decision-making under uncertainty and generalization of preference models, but hard to measure reliably. Metadata such as response times and interannotator agreement can serve as proxies for strength, but are often noisy and confounded. We propose ResponseRank to address the challenge of learning from noisy strength signals. Our method uses relative differences in proxy signals to rank responses to pairwise comparisons by their inferred preference strength. To control for systemic variation, we compare signals only locally within carefully constructed strata. This enables robust learning of utility differences consistent with strengthderived rankings while making minimal assumptions about the strength signal. Our contributions are threefold: (1) ResponseRank, a novel method that robustly learns preference strength by leveraging locally valid relative strength signals; (2) empirical evidence of improved sample efficiency and robustness across diverse tasks: synthetic preference learning (with simulated response times), language modeling (with annotator agreement), and RL control tasks (with simulated episode returns); and (3) the Pearson Distance Correlation (PDC), a novel metric that isolates cardinal utility learning from ordinal accuracy.

• Item Type: Conference or Workshop Item (Paper)
• EU Funded Grant Agreement Number: 101073307
• Faculties: Mathematics, Computer Science and Statistics > Computer Science > Artificial Intelligence and Machine Learning
• Subjects: 000 Computer science, information and general works > 004 Data processing computer science
• Place of Publication: San Diego, CA, USA
• DFG: Gefördert durch die Deutsche Forschungsgemeinschaft (DFG) - 251654672