Title: Quartet II: Enhancing Unbiased Gradient Estimation for Precise LLM Pre-Training in NVFP4

Abstract:

NVIDIA’s Blackwell GPUs introduce hardware support for the NVFP4 low-precision format, enabling the unprecedented capability to perform end-to-end fully quantized pre-training on large-scale language models. However, current quantized training techniques often compromise the representational power inherent to NVFP4. To achieve more accurate, unbiased gradient estimation through stochastic rounding (SR), these methods incur a significant accuracy penalty compared to traditional FP16 and FP8 training.

This study advances the state of the art in NVFP4 quantized training by introducing MS-EDEN, a new unbiased quantization routine tailored for micro-scaled formats. MS-EDEN reduces quantization error by more than half compared to stochastic rounding. We incorporate this technique into Quartet II, a novel quantization framework for linear layers that operates entirely in NVFP4. Our analytical results demonstrate that Quartet II delivers superior gradient estimation across all primary matrix multiplications during both forward and backward passes. Furthermore, the proposed method complements recent training optimizations designed specifically for NVFP4.

We empirically validate Quartet II through end-to-end LLM training on models with up to 1.9 billion parameters, processed over 38 billion tokens. The implementation includes kernels optimized for NVIDIA Blackwell GPUs, achieving a performance speedup of up to 4.2x relative to BF16. The source code is publicly available at https://github.com/IST-DASLab/Quartet-II .

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC