Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly

Kassner, Nora and Schütze, Hinrich (6. July 2020): Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly. The 58th Annual Meeting of the Association for Computational Linguistic, Seattle, USA, July 6 – 8, 2020. Stroudsburg, USA: Association for Computational Linguistics.

Abstract

Building on Petroni et al. (2019), we propose two new probing tasks analyzing factual knowledge stored in Pretrained Language Models (PLMs). (1) Negation. We find that PLMs do not distinguish between negated (“Birds cannot [MASK]”) and non-negated (“Birds can [MASK]”) cloze questions. (2) Mispriming. Inspired by priming methods in human psychology, we add “misprimes” to cloze questions (“Talk? Birds can [MASK]”). We find that PLMs are easily distracted by misprimes. These results suggest that PLMs still have a long way to go to adequately learn human-like factual knowledge.

• Item Type: Conference or Workshop Item (Paper)
• EU Funded Grant Agreement Number: 740516
• EU Projects: Horizon 2020 > ERC Grants > ERC Advanced Grant > ERC Grant 740516: NonSequeToR - Non-sequence models for tokenization replacement
• Research Centers: Center for Information and Language Processing (CIS)
• Subjects: 000 Computer science, information and general works > 000 Computer science, knowledge, and systems; 400 Language > 410 Linguistics
• Place of Publication: Stroudsburg, USA
• Annotation: https://arxiv.org/abs/1911.03343v3
• Language: English