Title: MX-SAFE: A Flexible Microscaling Format Ensuring Robustness for Both Training and Inference Through Dynamic Exponent and Mantissa Bit Allocation

Original: arXiv:2605.24391v2 Announce Type: replace-cross Abstract: As the demand for deep learning grows, cost reduction through quantization has become essential for both training and inference. In 2022, the Open Compute Project (OCP) consortium standardized narrow precision formats for deep learning, called the microscaling (MX) format. The MX format is a hardware-friendly dynamic quantization scheme that effectively reduces the data size by sharing an 8-bit exponent across multiple operands. The MX format can be categorized into two types with their own strengths: (i) MXINT which focuses on a high precision consisting only of mantissa bits and (ii) MXFP which focuses on a wider dynamic range by allowing local exponent bits. In this work, we present a versatile MXFP format, called MX-SAFE (MXSF in short), that adaptively uses two modes, i.e., a wider mantissa mode (FP8 E2M5) and a subnormal FP mode (FP5 E3M2), to support both training and direct-cast inference. Furthermore, we propose a tile-based block design to increase hardware efficiency by reducing the burden of re-quantization process during the training with the MXSF format. Owing to the use of the proposed MXSF format, 0.05%/11.1% and 3.55%/3.57% improvements in accuracy, on average, for inference/full-training compared to MXFP8 E2M5 and MXFP8 E4M3 are observed, respectively. Moreover, we present a training-inference accelerator that supports the MXSF format and it achieves similar accuracy to the BF16 baseline while using 24.9% less total energy consumption.

Rewritten: Title: MX-SAFE: A Versatile Microscaling Format with On-the-Fly Exponent and Mantissa Bit Allocation for Robust Training and Inference

Abstract: With the escalating requirements of deep learning, minimizing costs via quantization has emerged as a critical necessity for both inference and training workflows. In 2022, the Open Compute Project (OCP) consortium established standards for narrow-precision deep learning formats, known as microscaling (MX). This MX framework employs a hardware-efficient dynamic quantization method that shrinks data volume by distributing an 8-bit exponent across several operands. The MX architecture splits into two distinct variants, each offering unique advantages: MXINT, which prioritizes precision through mantissa-only bits, and MXFP, which expands dynamic range via local exponent bits. This study introduces MX-SAFE (abbreviated as MXSF), a flexible MXFP format that dynamically switches between a broad mantissa mode (FP8 E2M5) and a subnormal floating-point mode (FP5 E3M2) to accommodate both direct-cast inference and training tasks. Additionally, we introduce a tile-based block architecture designed to enhance hardware efficiency by alleviating the re-quantization workload during MXSF training. Leveraging the MXSF format yields average accuracy gains of 0.05% for inference and 11.1% for full training when compared against MXFP8 E2M5, and improvements of 3.55% and 3.57% respectively, relative to MXFP8 E4M3. Finally, we demonstrate a training-inference accelerator compatible with MXSF, which matches the accuracy of the BF16 baseline while reducing overall energy consumption by 24.9%.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC