Neural Semi-Markov Conditional Random Fields for Robust Character-Based Part-of-Speech Tagging

Kemos, Apostolos; Adel, Heike and Schütze, Hinrich (June 2019): Neural Semi-Markov Conditional Random Fields for Robust Character-Based Part-of-Speech Tagging. , Minneapolis, USA, 2. - 7. June 2019.

Abstract

Character-level models of tokens have been shown to be effective at dealing with withintoken noise and out-of-vocabulary words. However, they often still rely on correct token boundaries. In this paper, we propose to eliminate the need for tokenizers with an end-toend character-level semi-Markov conditional random field. It uses neural networks for its character and segment representations. We demonstrate its effectiveness in multilingual settings and when token boundaries are noisy: It matches state-of-the-art part-of-speech taggers for various languages and significantly outperforms them on a noisy English version of a benchmark dataset. Our code and the noisy dataset are publicly available at http://cistern.cis.lmu.de/semiCRF

• Item Type: Conference or Workshop Item (Paper)
• EU Funded Grant Agreement Number: 740516
• EU Projects: Horizon 2020 > ERC Grants > ERC Advanced Grant > ERC Grant 740516: NonSequeToR - Non-sequence models for tokenization replacement
• Form of publication: Preprint
• Research Centers: Center for Information and Language Processing (CIS)
• Subjects: 000 Computer science, information and general works > 000 Computer science, knowledge, and systems; 000 Computer science, information and general works > 004 Data processing computer science; 400 Language > 400 Language; 400 Language > 410 Linguistics
• Language: English