On the challenges and practices of reinforcement learning from real human feedback

Kaufmann, Timo ; Ball, Sarah; Beck, Jakob; Hüllermeier, Eyke und Kreuter, Frauke (22. September 2023): On the challenges and practices of reinforcement learning from real human feedback. First ECMLPKDD Workshop on Hybrid Human-Machine Learning and Decision Making, Turin, Italy, 22. September 2023. Machine Learning and Principles and Practice of Knowledge Discovery in Databases. Springer Cham. [Forthcoming]

Abstract

. Reinforcement learning from human feedback (RLHF) is a variant of reinforcement learning (RL) that does not require an engineered reward function but instead learns from human feedback. Due to its increasing popularity, various authors have studied how to learn an accurate reward model from only few samples, making optimal use of this feedback. Because of the cost and complexity of user studies, however, this research is often conducted with synthetic human feedback. Such feedback can be generated by evaluating behavior based on ground-truth rewards which are available for some benchmark tasks. While this setting can help evaluate some aspects of RLHF, it differs from practical settings in which synthetic feedback is not available. Working with real human feedback brings additional challenges that cannot be observed with synthetic feedback, including fatigue, inter-rater inconsistencies, delay, misunderstandings, and modality-dependent difficulties. We describe and discuss some of these challenges together with current practices and opportunities for further research in this paper.

• Dokumententyp: Konferenzbeitrag (Poster)
• Keywords: Reinforcement learning, RLHF, Human feedback
• Fakultät: Mathematik, Informatik und Statistik > Informatik > Künstliche Intelligenz und Maschinelles Lernen
• Themengebiete: 000 Informatik, Informationswissenschaft, allgemeine Werke > 004 Informatik
• ISBN: 978-3-031-74632-1
• Sprache: Englisch