Pixel-Level BPE for Auto-Regressive Image Generation

Razzhigaev A.; A. Voronov; Kaznacheev A.; Kusnetsov A.; Dimitrov D.; Panchenko A.

?

Pixel-Level BPE for Auto-Regressive Image Generation

P. 26–30.

Razzhigaev A., Voronov A., Kaznacheev A., Kusnetsov A., Dimitrov D., Panchenko A.

Pixel-level autoregression with Transformer models (Image GPT or iGPT) is one of the recent approaches to image generation that has not received massive attention and elaboration due to quadratic complexity of attention as it imposes huge memory requirements and thus restricts the resolution of the generated images. In this paper, we propose to tackle this problem by adopting Byte-Pair-Encoding (BPE) originally proposed for text processing to the image domain to drastically reduce the length of the modeled sequence. The obtained results demonstrate that it is possible to decrease the amount of computation required to generate images pixel-by-pixel while preserving their quality and the expressiveness of the features extracted from the model. Our results show that there is room for improvement for iGPT-like models with more thorough research on the way to the optimal sequence encoding techniques for images.

Language: English

Text on another site

Keywords: Generative pretrained transformer Transformer models

In book

Proceedings of the First Workshop on Performance and Interpretability Evaluations of Multimodal, Multipurpose, Massive-Scale Models (MMMPIE 2022)

Razzhigaev A., Voronov A., Kaznacheev A., Kuznetsov A., Dimitrov D., Panchenko A. International Conference on Computational Linguistics, 2022.

Prediction of protein-protein interactions using point transformer and spherical Convex Hull graphs

David Arteaga, Poptsova M., Computational and Structural Biotechnology Journal 2026 Vol. 31 P. 82–93

Accurate predictions and large-scale identification of protein-protein interactions (PPIs) are crucial for understanding their inherent biological mechanisms and protein functions in virtually all biological processes. Nowadays, graph-based deep learning models have made significant contributions in modeling proteins with physicochemical and geometric features. However, most of these models rely on conventional graph construction methods, such as ...

Added: December 22, 2025

Using Generative Pretrained Transformer-3 Models for Russian News Clustering and Title Generation tasks

Tikhonova M., Pisarevskaya D., Shavrina T. et al., Komp'juternaja Lingvistika i Intellektual'nye Tehnologii 2021 Vol. 20 P. 1214–1223

The paper presents a methodology for news clustering and news headline generation based on the zero-shot approach and minimal tuning of the RuGPT-3 architecture (Generative Pretrained Transformer 3 for Russian). The solution is presented in a competition for news clustering, headline selection and generation. The following approaches are described: 1) zero-shot unsupervised classification based on pairwise news perplexity: the ...

Added: September 22, 2023

Proceedings of the First Workshop on Performance and Interpretability Evaluations of Multimodal, Multipurpose, Massive-Scale Models (MMMPIE 2022)

Razzhigaev A., Voronov A., Kaznacheev A. et al., International Conference on Computational Linguistics, 2022.

Added: June 7, 2023

Continuous Prompt Tuning for Russian: How to Learn Prompts Efficiently with RuGPT3?

Tikhonova M., Konodyuk N., , in: Recent Trends in Analysis of Images, Social Networks and Texts. 10th International Conference, AIST 2021, Tbilisi, Georgia, December 16–18, 2021, Revised Selected Papers.: Springer, 2022. P. 30–40.

Adaptation to downstream tasks is a crucial part of the pre-trained language model (PLM) life cycle. Fine-tuning, traditionally used for this purpose, is an expensive procedure in terms of computation and memory. Dramatic growth of PLM capacities has led to the emergence of zero- and few-shot methods, which use natural language to describe tasks. Although ...

Added: October 31, 2022