Discrete and Soft Prompting for Multilingual Models

Zhao, Mengjie und Schütze, Hinrich (November 2021): Discrete and Soft Prompting for Multilingual Models. 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, November 2021. Moens, Marie-Francine; Huang, Xuanjing; Specia, Lucia und Yih, Scott Wen-tau (Hrsg.): In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Stroudsburg PA: Association for Computational Linguistics.

Abstract

It has been shown for English that discrete and soft prompting perform strongly in fewshot learning with pretrained language models (PLMs). In this paper, we show that discrete and soft prompting perform better than finetuning in multilingual cases: Crosslingual transfer and in-language training of multilingual natural language inference. For example, with 48 English training examples, finetuning obtains 33.74% accuracy in crosslingual transfer, barely surpassing the majority baseline (33.33%). In contrast, discrete and soft prompting outperform finetuning, achieving 36.43% and 38.79%. We also demonstrate good performance of prompting with training data in multiple languages other than English.

• Dokumententyp: Konferenzbeitrag (Paper)
• EU Funded Grant Agreement Number: 740516
• EU-Projekte: Horizon 2020 > ERC Grants > ERC Advanced Grant > ERC Grant 740516: NonSequeToR - Non-sequence models for tokenization replacement
• Fakultätsübergreifende Einrichtungen: Centrum für Informations- und Sprachverarbeitung (CIS)
• Themengebiete: 000 Informatik, Informationswissenschaft, allgemeine Werke > 000 Informatik, Wissen, Systeme ; 400 Sprache > 400 Sprache ; 400 Sprache > 410 Linguistik
• Ort: Stroudsburg PA
• Sprache: Englisch