Topic models with elements of neural networks: investigation of stability, coherence, and determining the optimal number of topics

PeerJ Computer Science. 2024. Vol. 10. P. 41.

Sergei Koltcov, Surkov A., Filippov V., Ignatenko V.

Topic modeling is a widely used instrument for the analysis of large text collections.
In the last few years, neural topic models and models with word embeddings have
been proposed to increase the quality of topic solutions. However, these models
were not extensively tested in terms of stability and interpretability. Moreover, the
question of selecting the number of topics (a model parameter) remains a challenging
task. We aim to partially fill this gap by testing four well-known and available to
a wide range of users topic models such as the embedded topic model (ETM),
Gaussian Softmax distribution model (GSM), Wasserstein autoencoders with Dirichlet
prior (W-LDA), and Wasserstein autoencoders with Gaussian Mixture prior (WTMGMM).
We demonstrate that W-LDA, WTM-GMM, and GSM possess poor stability
that complicates their application in practice. ETM model with additionally trained
embeddings demonstrates high coherence and rather good stability for large datasets,
but the question of the number of topics remains unsolved for this model. We also
propose a new topic model based on granulated sampling with word embeddings
(GLDAW), demonstrating the highest stability and good coherence compared to
other considered models. Moreover, the optimal number of topics in a dataset can
be determined for this model.

Keywords: coherence, topic modeling, Renyi entropy, Stability, optimal number of topics, word embeddings, neural topic models

Publication based on the results of: Innovative methods of data collection and analysis in the modeling of communicative behavior of Internet users and the development of respective technological solutions (2023)