Text Detoxification using Large Pre-trained Neural Models

Ch. 629. P. 7979–7996.

Dale D., Voronov A., Dementieva D., Logacheva V. K., Kozlova O., Semenov N., Panchenko A.

We present two novel unsupervised methods for eliminating toxicity in text. Our first method combines two recent ideas: (1) guidance of the generation process with small style-conditional language models and (2) use of paraphrasing models to perform style transfer. We use a well-performing paraphraser guided by style-trained language models to keep the text content and remove toxicity. Our second method uses BERT to replace toxic words with their non-offensive synonyms. We make the method more flexible by enabling BERT to replace mask tokens with a variable number of words. Finally, we present the first large-scale comparative study of style transfer models on the task of toxicity removal. We compare our models with a number of methods for style transfer. The models are evaluated in a reference-free way using a combination of unsupervised style transfer metrics. Both methods we suggest yield new SOTA results.

Keywords: Natural Language Processing (NLP)

In book

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Association for Computational Linguistics, 2021.