Learning Semantic Representations for Novel Words: Leveraging Both Form and Context

Schick, Timo and Schütze, Hinrich (January 2019): Learning Semantic Representations for Novel Words: Leveraging Both Form and Context. In: UNSPECIFIED

Abstract

Word embeddings are a key component of high-performing natural language processing (NLP) systems, but it remains a challenge to learn good representations for novel words on the fly, i.e., for words that did not occur in the training data. The general problem setting is that word embeddings are induced on an unlabeled training corpus and then a model is trained that embeds novel words into this induced embedding space. Currently, two approaches for learning embeddings ofnovel words exist: (i) learning an embedding from the novel word’s surface-form (e.g., subword n-grams) and (ii) learning an embedding from the context in which it occurs. In this paper, we propose an architecture that leverages both sources of information – surface-form and context – and show that itresults in large increases in embedding quality. Our architecture obtains state-of-the-art results on the Definitional Nonce and Contextual Rare Words datasets. As input, we only require an embedding set and an unlabeled corpus for training our architecture to produce embeddings appropriate for the induced embedding space. Thus, our model can easily be integrated into any existing NLP system and enhance its capability to handle novel words.

• Item Type: Conference or Workshop Item (Paper)
• EU Funded Grant Agreement Number: 740516
• EU Projects: Horizon 2020 > ERC Grants > ERC Advanced Grant > ERC Grant 740516: NonSequeToR - Non-sequence models for tokenization replacement
• Research Centers: Center for Information and Language Processing (CIS)
• Subjects: 000 Computer science, information and general works > 000 Computer science, knowledge, and systems; 000 Computer science, information and general works > 004 Data processing computer science; 400 Language > 400 Language; 400 Language > 410 Linguistics
• Language: English