Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly

Kassner, Nora und Schütze, Hinrich (6. Juli 2020): Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly. The 58th Annual Meeting of the Association for Computational Linguistic, Seattle, USA, July 6 – 8, 2020. Stroudsburg, USA: Association for Computational Linguistics.

Abstract

Building on Petroni et al. (2019), we propose two new probing tasks analyzing factual knowledge stored in Pretrained Language Models (PLMs). (1) Negation. We find that PLMs do not distinguish between negated (“Birds cannot [MASK]”) and non-negated (“Birds can [MASK]”) cloze questions. (2) Mispriming. Inspired by priming methods in human psychology, we add “misprimes” to cloze questions (“Talk? Birds can [MASK]”). We find that PLMs are easily distracted by misprimes. These results suggest that PLMs still have a long way to go to adequately learn human-like factual knowledge.

• Dokumententyp: Konferenzbeitrag (Paper)
• EU Funded Grant Agreement Number: 740516
• EU-Projekte: Horizon 2020 > ERC Grants > ERC Advanced Grant > ERC Grant 740516: NonSequeToR - Non-sequence models for tokenization replacement
• Fakultätsübergreifende Einrichtungen: Centrum für Informations- und Sprachverarbeitung (CIS)
• Themengebiete: 000 Informatik, Informationswissenschaft, allgemeine Werke > 000 Informatik, Wissen, Systeme ; 400 Sprache > 410 Linguistik
• Ort: Stroudsburg, USA
• Bemerkung: https://arxiv.org/abs/1911.03343v3
• Sprache: Englisch