Title: Moving Beyond Discreteness: A Sample Complexity Study of the Straight-Through Estimator in 1-Bit Quantization

Abstract: The training of quantized neural networks is complicated by the discrete and non-differentiable characteristics of the optimization landscape. To overcome this hurdle, the straight-through estimator (STE) has emerged as the predominant heuristic, enabling backpropagation through discrete steps by employing surrogate gradients that are biased but functionally valid. Despite its widespread use, the theoretical underpinnings of STE are not well understood; previous analyses have primarily concentrated on generalization error, often relying on the assumption of infinite training data. This study introduces the initial sample complexity analysis for STE within the domain of neural network quantization. Our findings emphasize the pivotal influence of sample size on the efficacy of STE, a crucial factor overlooked in prior research. By examining the quantization-aware training of a two-layer neural network characterized by binary weights and activations, we establish sample complexity bounds dependent on data dimensionality. These bounds ensure that STE-based optimization converges to the global minimum, applicable to both ergodic and non-ergodic frameworks. Furthermore, we identify a distinctive recurrence behavior in the STE-gradient method when label noise is present, wherein the iterates consistently diverge from and subsequently return to the optimal binary weights. Empirical evaluations reveal that while STE is ineffective for general non-Gaussian data, its performance can be reinstated by applying normalization, thereby highlighting its practical significance in achieving effective quantization.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC