Automatic dependency parsing of a learner English corpus REALEC

NRU HSE , 2017.

Lyashevskaya O., Пантелеева И. М.

The paper presents a Universal Dependencies (UD) annotation scheme for a learner English corpus. The REALEC dataset consists of essays written in English by Russian-speaking university students in the course of general English. The essays are a part of students' preparation for the independent final examination similar to the international English exam. While adjusting existing dependency parsing tools to a learner data, one has to take into account to what extent students' mistakes provoke errors in the parser output. The ungrammatical and stylistically inappropriate utterances may challenge parsers' algorithms trained on grammatically appropriate written texts. In our experiments, we compared the output of the dependency parser UDpipe (trained on UD-English 2.0) with the results of manual parsing, placing a particular focus on parses of ungrammatical English clauses. We show how mistakes made by students influence the work of the parser. Overall, UDpipe performed reasonably well (UAS 92.9, LAS 91.7). The following cases cause the errors in automatic annotation a) incorrect detection of a head, b) incorrect detection of the relation type, as well as c) both. We propose some solutions which could improve the automatic output and thus make the assessment of syntactic complexity more reliable.

Keywords: учебный корпус, английский язык как иностранный, learner corpus, universal dependencies, универсальные зависимости, dependency annotation of learner treebank, evaluation of parser quality, L2 English, синтаксическая разметка корпуса, разметка синтаксических зависимостей, синтаксическая разметка учебного корпуса, оценка качества синтаксического парсинга

Publication based on the results of: Лексикологические исследования на базе учебного корпуса REALEC (Learner corpus REALEC: Lexicological observations) (2016)