Title: WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantization

Abstract:

This study addresses the challenge of transforming standard dense linear layers into low-precision formats. We conduct an information-theoretic (IT) analysis of the fundamental trade-off between the resulting compressed data length and the discrepancy in the layer's output. Our investigation reveals that the widely used GPTQ algorithm can exhibit an arbitrarily significant deviation from the theoretical IT limit. To bridge this gap, we introduce a new method called "WaterSIC." This approach demonstrates a uniform rate gap of no more than 0.255 bits to the IT limit, regardless of the input activation covariance matrices. The core innovation of WaterSIC lies in its ability to assign distinct quantization rates to individual columns (in-features) of the weight matrix, effectively emulating the classical information-theoretic strategy known as "waterfilling." When applied to the Llama and Qwen large language model families, WaterSIC achieves new state-of-the-art results across all quantization levels ranging from 1 to 4 bits. The source code for this work is publicly accessible at https://github.com/egorlifar/watersic.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC