Using Generative Pretrained Transformer-3 Models for Russian News Clustering and Title Generation tasks

Komp'juternaja Lingvistika i Intellektual'nye Tehnologii. 2021. Vol. 20. P. 1214–1223.

Tikhonova M., Pisarevskaya D., Shavrina T., Shliazhko O.

The paper presents a methodology for news clustering and news headline generation based on the zero-shot
approach and minimal tuning of the RuGPT-3 architecture (Generative Pretrained Transformer 3 for Russian). The
solution is presented in a competition for news clustering, headline selection and generation.
The following approaches are described: 1) zero-shot unsupervised classification based on pairwise news
perplexity: the method requires no training or model fine-tuning and yields 0.7 F1-measure.
2) fine-tuning: news headlines generation with the best result 0.292 ROUGE and 0.596 BLEU.

Keywords: text generation, Generative pretrained transformer, Evaluation track, text clustering, ruGPT-3