The smaller the better? Heterogeneity of corpus, training size, and morphological tagging

P. 1091–1108.

Lyashevskaya O., Ostyakova L., Сальников Е. А., Семенова О. А.

Orthographic and morphological heterogeneity of historical texts in pre-modern Slavic causes many difficulties in pos- and morphological tagging. Existing approaches to these tasks show state-of-the-art results without normalization, but they are still very sensitive to the properties of training data such as genre and origin. In this paper, we investigate to what extent the heterogeneity and size of the training corpus influence the quality of pos tagging and morphological analysis. We observe that UDpipe trained on different parts of the Middle Russian corpus demonstrates a boost in accuracy when using less training data. We resolve this paradox by analyzing the distribution of pos-tags and short words across subcorpora.

Keywords: part of speech tagging, морфологическая разметка, частеречная разметка, full morphological tagging, historical data, corpus size, corpus data homogeneity, автоматическая обработка исторических текстов

In book

Компьютерная лингвистика и интеллектуальные технологии: По материалам ежегодной международной конференции «Диалог» (Москва, 17 июня — 20 июня 2020 г.). Дополнительный том материалов

M.: ., 2020.