Comparing models of morpheme analysis for Russian words based on machine learning

P. 104–113.

Bolshakova E. I., Sapin A.

The paper reports on the experimental comparison of several machine learning models proposed in recent years for automatic morpheme segmentation of Russian words, including conditional random fields (CRF), sequence-to-sequence neural network (Seq2seq) , convolutional neural network (CNN) model, as well as a new model we have developed with the aid of gradient boosted decision trees (GBDT). For more complete research, in our experiments we have also evaluated the semi-supervised method of Morfessor. All the morpheme analysis models being compared are briefly described in the paper, some of them perform only segmentation of words into morphs, the other produce segmentation with classification of resulted morphs. Since for Russian language linguistics rules for splitting words into morphs (and also the classification of some morphs) may differ, the experiments were performed for two data sets differing in labeling, which are obtained respectively from CrossLexica's dictionary and Tikhonov's dictionary. The experimental evaluation has shown that two best models of morpheme segmentation with classification, namely GBDT and CNN models have comparable quality, giving about 86-94% of word-level accuracy.

Keywords: computational linguistics, morphological segmentation

In book

Computational Linguistics and Intellectual Technologies Papers from the Annual International Conference “Dialogue” (2019)

Issue 18. , M.: Russian State University for the Humanitie, 2019.