Title: Model-Preserving Adaptive Rounding

Abstract:

Quantization aims to generate compressed models that replicate the output distribution of the original architecture as closely as possible. To make this process computationally feasible, standard quantization techniques typically minimize the immediate activation error of individual layers, treating it as a surrogate for the overall end-to-end error. However, this approach overlooks the influence of subsequent layers, rendering it an inadequate proxy for total model performance.

In this study, we present Yet Another Quantization Algorithm (YAQA), an adaptive rounding method that directly optimizes for error at the network’s final output. YAQA is supported by a series of theoretical contributions, marking the first provision of end-to-end error bounds for quantization algorithms. Specifically, we analyze the convergence speed of adaptive rounding methods by examining their Hessian approximation structures. We demonstrate that the end-to-end error can be constrained by the cosine similarity between the Hessian approximation and the true Hessian. This insight enables the use of a natural Kronecker-factored approximation, paired with near-optimal Hessian sketches.

Theoretical analysis proves that YAQA outperforms GPTQ and LDLQ. Empirical results confirm this advantage, showing an error reduction of approximately 30% compared to these methods. Notably, YAQA achieves lower error rates than quantization-aware training. These improvements translate into state-of-the-art performance on downstream tasks, all without introducing any additional inference overhead.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC