Grammar in Language Models: BERT Study

NRU HSE , 2023. No. 115.

Chistyakova K., Kazakova Tatiana

The problem of language models’ interpretation is extensively inspected, but no universal answers have been found. Our study offers to combine widely accepted probing methods with a novel approach to a neural network under investigation. We propose to break grammatical forms on the pre-training step in order to get two "sibling" models, as it casts some light on how different linguistic features are encoded and distributed across the neural language architecture. Project repository: https://github.com/skipdividedd/bert_grammar

Keywords: языковые модели, BERT, Probing, Пробинг, BERT, transformers, трансформеры, language models