LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Framework

Zhao, Mengjie; Mi, Fei; Wang, Yasheng; Li, Minglei; Jiang, Xin; Liu, Qun and Schütze, Hinrich (July 2022): LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Framework. NAACL 2022, Seattle, United States, July 2022. Carpuat,, Marine; de Marneffe, Marie-Catherine and Meza Ruiz, Ivan Vladimir (eds.) : In: Findings of the Association for Computational Linguistics: NAACL 2022, Stroudsburg, PA: Association for Computational Linguistics (ACL). pp. 675-692

Abstract

Vast efforts have been devoted to creating high-performance few-shot learners, i.e., large-scale pretrained language models (PLMs) that perform well with little downstream task training data. Training PLMs has incurred significant cost, but utilizing the few-shot learners is still challenging due to their enormous size. This work focuses on a crucial question: How to make effective use of these few-shot learners? We propose LMTurk, a novel approach that treats few-shotlearners as crowdsourcing workers. The rationale is that crowdsourcing workers are in fact few-shot learners: They are shown a few illustrative examples to learn about a task and then start annotating. LMTurk employs few-shot learners built upon PLMs as workers. We show that the resulting annotations can be utilized to train models that solve the task well and are small enough to be deployable in practical scenarios. Active learning is integrated into LMTurk to reduce the amount of queries made to PLMs, minimizing the computational cost of running PLM inference passes. Altogether, LMTurk is an important step towards making effective use of current PLMs.

• Item Type: Conference or Workshop Item (Paper)
• EU Funded Grant Agreement Number: 740516
• EU Projects: Horizon 2020 > ERC Grants > ERC Advanced Grant > ERC Grant 740516: NonSequeToR - Non-sequence models for tokenization replacement
• Research Centers: Center for Information and Language Processing (CIS)
• Subjects: 000 Computer science, information and general works > 000 Computer science, knowledge, and systems; 400 Language > 400 Language; 400 Language > 410 Linguistics
• Place of Publication: Stroudsburg, PA
• Annotation: ISBN 978-1-955917-76-6
• Language: English