?
MARS: Masked Automatic Ranks Selection in Tensor Decompositions
P. 3718–3732.
Tensor decomposition methods have proven effective in various applications, including compression and acceleration of neural networks. At the same time, the problem of determining optimal decomposition ranks, which present the crucial parameter controlling the compressionaccuracy trade-off, is still acute. In this paper, we introduce MARS - a new efficient method for the automatic selection of ranks in general tensor decompositions. During training, the procedure learns binary masks over decomposition cores that “select” the optimal tensor structure. The learning is performed via relaxed maximum a posteriori (MAP) estimation in a specific Bayesian model and can be naturally embedded into the standard neural network training routine. Diverse experiments demonstrate that MARS achieves better results compared to previous works in various tasks.
In book
Vol. 206. , Valencia: PMLR, 2023.
Aleksei Samarin, Nazarenko A., Kotenko E. et al., Machine Learning and Knowledge Extraction 2026 Vol. 8 No. 8 P. 1–26
This paper presents a novel method for pruning deep neural networks based on the concept of flow, derived from the continuous modeling of signal propagation across layers. We derive flow functions for fully connected, convolutional, and self-attention architectures, and we propose a new iterative pruning algorithm, Iterative Flow-Aware Pruning (IFAP), that leverages these measures to ...
Added: September 19, 2026
Perminov Andrew Igorevich, / Series Computer Science "arxiv.org". 2025.
Matrix multiplication optimization remains a fundamental challenge in computational mathematics. This work introduces a novel approach that discovers matrix multiplication schemes whose coefficients are restricted to the set {−1,0,1} (denoted ZT), minimizing naive additive complexity for efficient hardware implementation. The core of the method is a GPU-accelerated meta flip graph algorithm that maintains ternary safety ...
Added: September 14, 2026
Perminov Andrew Igorevich, / Series Computer Science "arxiv.org". 2026.
An open-source C++ framework for discovering fast matrix multiplication schemes using the flip graph approach is presented. The framework supports multiple coefficient rings -- binary (Z2), modular ternary (Z3) and integer ternary (ZT={−1,0,1}) -- and implements both fixed-dimension and meta-dimensional search operators. Using efficient bit-level encoding of coefficient vectors and OpenMP parallelism, the tools enable ...
Added: September 14, 2026
Perminov Andrew Igorevich, / Series Computer Science "arxiv.org". 2026.
This paper presents new results for fast matrix multiplication in small formats obtained by combining the meta flip graph framework with the serendipitous product construction. The framework has been extended to support all 680 rectangular formats with dimensions up to 16×16×16. Compared to the previous state of the art, ranks are improved for 207 formats. ...
Added: September 14, 2026
Grishina E., Gorbunov M., Rakhuba M., , in: Findings of the Association for Computational Linguistics: ACL 2025.: Association for Computational Linguistics, 2025. P. 26937–26949.
Large language models (LLMs) demonstrate impressive results in natural language processing tasks but require a significant amount of computational and memory resources. Structured matrix representations are a promising way for reducing the number of parameters of these models. However, it seems unrealistic to expect that weight matrices of pretrained models can be accurately represented by ...
Added: September 4, 2025
Kodryan M., Kropotov D., Vetrov D., / Series QTNML 2020 "First Workshop on Quantum Tensor Networks in Machine Learning, NeurIPS 2020". 2020.
Tensor decomposition methods have recently proven to be efficient for compressing and accelerating neural networks. However, the problem of optimal decomposition structure determination is still not well studied while being quite important. Specifically, decomposition ranks present the crucial parameter controlling the compression-accuracy trade-off. In this paper, we introduce MARS - a new efficient method for ...
Added: February 5, 2021
Shitov Y., Acta Mathematica 2019 Vol. 222 No. 2 P. 363–379
The rank of tensors is not additive with respect to the direct sum. ...
Added: November 22, 2020
Zykin A. I., Ballet S., Designs, Codes and Cryptography 2019 Vol. 87 P. 517–525
We obtain new uniform bounds for the symmetric tensor rank of multiplication in finite extensions of any finite field F_p or F_{p^2} where p denotes a prime number ≥5. In this aim, we use the symmetric Chudnovsky-type generalized algorithm applied on sufficiently dense families of modular curves defined over F_{p_2} attaining the Drinfeld–Vladuts bound and on the descent of these families to ...
Added: May 12, 2020
Vetrov D., Izmailov P., Maddox W. et al., , in: Proceedings of the 35th Uncertainty in Artificial Intelligence Conference (UAI-2019).: [б.и.], 2019. P. 1–11.
Bayesian inference was once a gold standard for learning with neural networks, providing accurate full predictive distributions and well calibrated uncertainty. However, scaling Bayesian inference techniques to deep neural networks is challenging due to the high dimensionality of the parameter space. In this paper, we construct low-dimensional subspaces of parameter space, such as the first ...
Added: January 14, 2020
Vorontsov K. V., Kochedykov D., Apishev M. et al., IEEE Computer Society, 2017.
Topic modelling is an area of text mining that has been actively developed in the last 15 years. A probabilistic topic model extracts a set of hidden topics from a collection of text documents. It defines each topic by a probability distribution over words and describes each document with a probability distribution over topics. In ...
Added: December 6, 2019
Bazenkov N., Goubko M., , in: Proceedings 16th Russian Conference on Artificial Intelligence (RCAI 2018)Issue 934.: Cham: Springer, 2018.
For modern energy markets it is typical to use dynamic real-time pricing schemes even for residential customers. Such schemes are expected to stimulate rational energy consumption by the end customers, provide peak shaving and overall energy efficiency. But under dynamic pricing planning a household’s energy consumption becomes complicated. So automated planning of household appliances is ...
Added: December 2, 2019
Grachev A., Ignatov D. I., Savchenko A., Applied Soft Computing Journal 2019 Vol. 79 P. 354–362
Recurrent neural networks have proved to be an effective method for statistical language modeling. However, in practice their memory and run-time complexity are usually too large to be implemented in real-time offline mobile applications. In this paper we consider several compression techniques for recurrent neural networks including Long–Short Term Memory models. We make particular attention ...
Added: June 12, 2019
Shitov Y., Linear Algebra and its Applications 2018 Vol. 544 P. 299–305
An n×n matrix A is called permutative if the rows of A are distinct permutations of a family of n distinct elements. For all n⩾3, we show that the minimal rank of a non-negative permutative matrix equals 3. The minimal rank of a generic permutative n×n matrix equals the smallest integer r such that r!⩾n. ...
Added: January 30, 2019
Izmailov P., Novikov A., Kropotov D., , in: Proceedings of Machine Learning Research. Proceedings of The International Conference on Artificial Intelligence and Statistics (AISTATS 2018).: [б.и.], 2018. P. 726–735.
We propose a method (TT-GP) for approximate inference in Gaussian Process (GP) models. We build on previous scalable GP research including stochastic variational inference based on inducing inputs, kernel interpolation, and structure exploiting algebra. The key idea of our method is to use Tensor Train decomposition for variational parameters, which allows us to train GPs ...
Added: December 10, 2018
Shitov Y., SIAM Journal on Applied Algebra and Geometry 2018 Vol. 2 No. 3 P. 428–443
The rank and symmetric rank of a symmetric tensor may differ. ...
Added: September 26, 2018
Khrulkov V., Novikov A., Oseledets I., , in: Proceedings of the 6th International Conference on Learning Representations (ICLR 2018).: [б.и.], 2018. P. 1–12.
Added: February 4, 2018
Izmailov P., Novikov A., Kroptov D., / Series arXiv "math". 2017.
We propose a method (TT-GP) for approximate inference in Gaussian Process (GP) models. We build on previous scalable GP research including stochastic variational inference based on inducing inputs, kernel interpolation, and structure exploiting algebra. The key idea of our method is to use Tensor Train decomposition for variational parameters, which allows us to train GPs ...
Added: October 20, 2017
A. G. Rassadin, A. V. Savchenko, , in: CEUR Workshop ProceedingsVol. 1901: Proceedings of the International conference Information Technology and Nanotechnology. Session Image Processing, Geoinformation Technology and Information Security.: CEUR-WS, 2017. P. 207–213.
In this paper, we consider the problem of insufficient runtime and memory space complexities of deep convolutional neural networks for visual emotion recognition. A survey of recent compression methods and efficient neural networks architectures is provided. We experimentally compare the computational speed and memory consumption during the training and the inference stages of such methods ...
Added: October 17, 2017
Губко М. В., Kuznetsov S., Neznanov A. et al., IFAC-PapersOnLine 2016 Vol. 49 No. 32 P. 24–29
In coming years residential consumers will face real-time electricity tariffs with energy prices varying day to day, and effective energy saving will require automation - a recommender system, which learns consumer's preferences from her actions. A consumer chooses a scenario of home appliance use to balance her comfort level and the energy bill. We propose ...
Added: January 24, 2017