Logo Logo
Switch Language to German

Kemos, Apostolos; Adel, Heike and Schütze, Hinrich (June 2019): Neural Semi-Markov Conditional Random Fields for Robust Character-Based Part-of-Speech Tagging. , Minneapolis, USA, 2. - 7. June 2019. [PDF, 369kB]


Character-level models of tokens have been shown to be effective at dealing with withintoken noise and out-of-vocabulary words. However, they often still rely on correct token boundaries. In this paper, we propose to eliminate the need for tokenizers with an end-toend character-level semi-Markov conditional random field. It uses neural networks for its character and segment representations. We demonstrate its effectiveness in multilingual settings and when token boundaries are noisy: It matches state-of-the-art part-of-speech taggers for various languages and significantly outperforms them on a noisy English version of a benchmark dataset. Our code and the noisy dataset are publicly available at http://cistern.cis.lmu.de/semiCRF

Actions (login required)

View Item View Item