Title: AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization

Abstract:

While Mixture-of-Experts (MoE) architectures enhance model capacity via sparse expert activation, their practical deployment is often hindered by memory constraints, as all expert parameters must be stored in memory. Mixed-precision quantization offers a solution by reducing this footprint through the assignment of varying bit-widths to different experts. However, current methods generally depend on calibration data to gauge expert importance and decide on bit allocation. For leading MoE large language models (LLMs), the original training data—and consequently the true training distribution—is proprietary and unavailable. This forces the use of imperfect calibration sets, which can distort estimates of expert utilization and result in suboptimal bit allocation.

Addressing the significant variability in quality across experts in modern MoE models, and leveraging the proven ability of Heavy-Tailed Self-Regularization (HT-SR) theory to predict neural network performance without training or testing data, we introduce AlphaQ. This is a calibration-free bit-allocation strategy for MoE quantization. Guided by HT-SR theory, AlphaQ operates on the premise that experts exhibiting stronger heavy-tailed weight spectra are generally better trained and thus merit higher bit-widths, whereas those with less pronounced heavy-tailed structures can undergo more aggressive quantization. To implement this, AlphaQ measures the spectral heavy-tailedness of each expert and resolves a budget-constrained optimization problem designed to minimize total quantization error within a specified global bit budget.

Experimental results across multiple MoE models demonstrate that AlphaQ consistently surpasses calibration-based baselines when compared under identical bit budgets. Specifically, on the Qwen1.5-MoE model, AlphaQ maintains near full-precision accuracy while using an average expert precision of just 3.5 bits, achieving over 4$\times$ memory compression. The code for AlphaQ is publicly available at https://github.com/Superone77/AlphaQ.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC