An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers

Hofmann, Valentin; Schütze, Hinrich und Pierrehumbert, Janet B. (May 2022): An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers. 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland, May 22-27, 2022. Muresan, Smarandakov; Nakov, Preslav und Villavicencio, Aline (eds.) : In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Stroudsburg, PA: Association for Computational Linguistics. pp. 385-393

Abstract

We introduce FLOTA (Few Longest Token Approximation), a simple yet effective method to improve the tokenization of pretrained language models (PLMs). FLOTA uses the vocabulary of a standard tokenizer but tries to preserve the morphological structure of words during tokenization. We evaluate FLOTA on morphological gold segmentations as well as a text classification task, using BERT, GPT-2, and XLNet as example PLMs. FLOTA leads to performance gains, makes inference more efficient, and enhances the robustness of PLMs with respect to whitespace noise.

• Item Type: Conference or Workshop Item (Paper)
• EU Funded Grant Agreement Number: 740516
• EU Projects: Horizon 2020 > ERC Grants > ERC Advanced Grant > ERC Grant 740516: NonSequeToR - Non-sequence models for tokenization replacement
• Research Centers: Center for Information and Language Processing (CIS)
• Subjects: 000 Computer science, information and general works > 000 Computer science, knowledge, and systems; 400 Language > 400 Language; 400 Language > 410 Linguistics
• Place of Publication: Stroudsburg, PA
• Language: English