Comparing Neural Lexical Models of a Classic National Corpus and a Web Corpus: The Case for Russian

P. 47–58.

Kutuzov A. B., Kuzmenko E.

In this paper we compare the Russian National Corpus to a larger Russian web corpus composed in 2014; the assumption behind our work is that the National corpus, being limited by the texts it contains and their proportions, presents lexical contexts (and thus meanings) which are different from those found ‘in the wild’ or in a language in use.

To do such a comparison, we used both corpora as training sets to learn vector word representations and found the nearest neighbors or associates for all top-frequency nominal lexical units. Then the difference between these two neighbor sets for each word was calculated using the Jaccard similarity coefficient. The resulting value is the measure of how much the meaning of a given word is different in the language of web pages from the Russian language in the National corpus. About 15% of words were found to acquire completely new neighbors in the web corpus.

In this paper, the methodology of research is described and implications for Russian National Corpus are proposed. All experimental data are available online.

 

Keywords: corpus linguistics, computational linguistics, Russian National Corpus, word co-occurrence, semantic similarity of words, deep learning, corpora comparison, web as corpus, word2vec

Publication based on the results of: Corpus studies of language variation: from deviations to linguistic norm (2015)

In book

Computational Linguistics and Intelligent Text Processing, Lecture Notes in Computer Science

Vol. 9041. , Springer, 2015.