AutoJudge: Judge Decoding Without Manual Annotation

P. 94605–94642.

Roman Garipov, Fedor Velikonivtsev, Ivan Ermakov, Ruslan Svirschevski, Vage Egiazarian, Max Ryabinin

We introduce AutoJudge, a method that accelerates large language model (LLM) inference with task-specific lossy speculative decoding. Instead of matching the original model output distribution token-by-token, we identify the generated tokens that affect the downstream quality of the response, relaxing the distribution match guarantee so that the "unimportant" tokens can be generated faster.Our approach relies on a semi‑greedy search algorithm to test which of the mismatches between target and draft models should be corrected to preserve quality and which ones may be skipped.We then train a lightweight classifier based on existing LLM embeddings to predict, at inference time, which mismatching tokens can be safely accepted without compromising the final answer quality.We evaluate AutoJudge with multiple draft/target model pairs on mathematical reasoning and programming benchmarks, achieving significant speedups at the cost of a minor accuracy reduction. Notably, on GSM8K with the Llama 3.1 70B target model, our approach achieves up to speedup \textit{over speculative decoding} at the cost of a drop in accuracy.When applied to the LiveCodeBench benchmark, AutoJudge automatically detects programming-specific important tokens, accepting tokens per speculation cycle at a  drop in Pass@1. Our approach requires no human annotation and is easy to integrate with modern LLM inference frameworks.

Keywords: LLM, Lossy Speculative Decoding

In book

39th Conference on Neural Information Processing Systems (NeurIPS 2025)

NeurIPS, 2025.