MinMAE calibration method for convolutional neural network quantization

PeerJ Computer Science. 2026. Vol. 12. Article 3724.

Vasilev A., Kapitanov A., Roman Solovyev, Dmitry Telpukhov, Stempkovskiy A., Kustov A., Prutyanov V., Romanov A.

This article introduces MinMAE, a novel activation calibration method for Post-Training Quantization (PTQ) that significantly reduces accuracy loss in Convolutional Neural Networks (CNN). Motivated by the need for high-fidelity quantization without costly retraining, MinMAE directly minimizes the Mean Absolute Error (MAE) between original and dequantized activations, making it robust to outliers that degrade standard methods. We evaluated MinMAE against absmax and mean+3std on YOLOv8n and YOLOv8s models. For 8-bit quantization on YOLOv8n, MinMAE achieved 8.4% mean Average Precision (mAP) loss, whereas standard methods led to drops of up to 14.4%. On the larger YOLOv8s model, MinMAE also demonstrated superior performance, reducing loss from 7.2% to just 5.2%. While this approach requires a higher one-time calibration time, the substantial accuracy gains establish MinMAE as a highly effective and broadly applicable strategy for improving the stability and performance of quantized CNNs.

Keywords: quantization, object detection, neural network