Title: FLARE: A Diffusion-Based Approach for Hybrid Language Models

Abstract:

While autoregressive (AR) large language models (LLMs) have secured widespread practical utility, their reliance on sequential decoding creates a significant hurdle for low-latency applications. Current efforts to optimize inference efficiency generally follow two paths: lowering the computational cost per model call via efficient architectures, and minimizing serial decoding steps through parallel generation. Hybrid attention architectures tackle the former, whereas diffusion language models (dLLMs) address the latter by employing iterative parallel denoising. Merging these strengths, however, proves difficult. Converting AR models to dLLMs frequently compromises the ability to maintain seed-checkpoint integrity, while the recurrent states and masking requirements inherent to hybrid attention comp both diffusion training and deployment.

To address these challenges, we introduce FLARE, a comprehensive framework designed for converting hybrid-attention LLMs. Our investigation reveals that the quality of transfer data is the most critical factor in preserving model capabilities, surpassing the influence of loss formulations and attention-mask configurations. FLARE integrates a unified inference mechanism, hardware-optimized kernels, and a token-balanced objective that serves both AR and diffusion paradigms. This allows a single checkpoint to facilitate both AR-style verified decoding and diffusion-style parallel denoising.

Beginning with robust AR checkpoints and utilizing minimal post-training data, FLARE performs competitively against top-tier open-source dLLMs across various model sizes. Furthermore, it provides consistent throughput improvements over open-source dLLM baselines during single-GPU concurrent serving. These findings indicate that the practical limitations of dLLMs stem not only from decoding algorithms but also from transfer data quality and the training inefficiencies associated with current block-diffusion objectives. This underscores the need for a holistic design approach that jointly considers data, objectives, architectures, and inference systems.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC