Title: Can Uncertainty Be Maintained Through Compression? A Conformal Prediction-Based Benchmark for Quantized and Sparse Large Language Models

Abstract:

To mitigate the high costs associated with deploying large language models (LLMs), practitioners widely employ compression strategies like quantization and pruning. Current assessment methods, however, tend to prioritize the retention of accuracy above all else. In contexts where safety is paramount, a model’s capacity to accurately gauge its own uncertainty is just as critical. This study investigates whether such compression techniques compromise this vital capability. By applying conformal prediction—a robust, distribution-free method for measuring uncertainty—we evaluated 12 LLMs across five distinct NLP tasks under various compression settings. Our findings highlight three key insights: (I) accuracy and uncertainty reliability are often decoupled by compression; (II) larger architectures are significantly better at absorbing uncertainty introduced by compression compared to their smaller counterparts; and (III) increases in uncertainty tend to occur in sharp, threshold-like jumps rather than gradual increments. These outcomes indicate that relying solely on accuracy metrics is inadequate for determining if compressed LLMs are ready for deployment. Consequently, incorporating uncertainty-aware benchmarking should become a standard practice within model compression workflows.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC