Logo Logo
Switch Language to German
Kemos, Apostolos; Adel, Heike; Schütze, Hinrich (June 2019): Neural Semi-Markov Conditional Random Fields for Robust Character-Based Part-of-Speech Tagging. UNSPECIFIED, 2. - 7. June 2019, Minneapolis, USA.


Character-level models of tokens have been shown to be effective at dealing with withintoken noise and out-of-vocabulary words. However, they often still rely on correct token boundaries. In this paper, we propose to eliminate the need for tokenizers with an end-toend character-level semi-Markov conditional random field. It uses neural networks for its character and segment representations. We demonstrate its effectiveness in multilingual settings and when token boundaries are noisy: It matches state-of-the-art part-of-speech taggers for various languages and significantly outperforms them on a noisy English version of a benchmark dataset. Our code and the noisy dataset are publicly available at http://cistern.cis.lmu.de/semiCRF