Title: Quantized Reasoning Models Perceive a Need for Extended Thought, Yet It Is Unnecessary

Abstract:

While Post-Training Quantization (PTQ) is a standard technique for efficiently deploying large language models, its specific impact on reasoning capabilities remains poorly characterized. Our analysis across domains including mathematics, coding, and scientific question answering reveals that aggressive PTQ not only diminishes accuracy but also triggers an increase in the length of chain-of-thought (CoT) reasoning. Contrary to the assumption that these models fail to find solutions, we demonstrate that in as many as 52% of cases where quantized models fail, they actually identify the correct answer during intermediate steps but fail to present it as the final output.

To investigate the mechanisms behind this phenomenon, known as "overthinking," we calculated the token-level Kullback-Leibler (KL) divergence between the output distributions of quantized and full-precision models. We observed a strong correlation between positions exhibiting high KL divergence and those with high next-token entropy. At these specific junctures, quantized models disproportionately select overthinking markers, such as "wait," "but," and "alternatively."

We propose a simple, training-free intervention: applying a logit penalty to a curated list of these overthinking markers. This approach reduces CoT length by 12–23% while maintaining or enhancing accuracy. This result holds consistent across five models (ranging from 1.5B to 32B parameters), three distinct quantization methods, and five different benchmarks. The strategy establishes a favorable Pareto frontier, offering a better trade-off between accuracy and reasoning cost compared to penalties applied to other token groups. Furthermore, this method successfully reduces overthinking errors generated by quantized models by up to 58%.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC