?
LiteKV: A KV cache compression method for efficient inference on resource-constrained devices
Transformer-based Large Language Models (LLMs) rely on Key-Value (KV) cache to accelerate autoregressive decoding, but memory and compute requirements still grow with context length, limiting deployment on resource-constrained devices. Existing training-free KV cache compression methods mainly rely on attention-driven retention heuristics, which may under preserve structurally informative tokens under tight cache budgets, especially when such tokens are not strongly highlighted by local or cumulative attention signals. As a result, long-range evidence and semantic continuity can be weakened in long-context inference. We propose LiteKV, a training-free KV cache management strategy that addresses this limitation with a three-ring cache structure and a Semantic Shift Awareness (SSA) mechanism. LiteKV maintains a recent window, a heavy-hitter region, and an overview zone, while SSA introduces a semantic-momentum signal to complement attention-based retention and improve the preservation of transition-relevant tokens. We evaluate LiteKV on Qwen 3, Llama 3.1, and Mistral 7B models against Full KV, StreamingLLM, H2O, and SnapKV on several tasks. On LongPPL over 20k tokens from WikiText-103, LiteKV reduces perplexity from 12.74 for H2O to 10.68, corresponding to a 16.2% reduction. In the Needle-in-a-Haystack task, LiteKV’s retrieval accuracy matches the Full KV baseline in several segments and remains substantially stronger than H2O and StreamingLLM in entity-specific retrieval. LiteKV also remains robust on L-Eval benchmark under aggressive compression. For example, on Qwen3-30B with a 3.125% cache budget, LiteKV retains 83.5% of the Full KV score on meeting summarization task and 99.1% on review summarization task. These results suggest a favorable memory–quality trade-off for long-context inference across multiple model families.