Title: d2: Enhancing Reasoning Capabilities in Diffusion Language Models Through Trajectory Likelihood Estimation

Diffusion language models (DLMs) have demonstrated strong results in text generation; however, leveraging reinforcement learning to boost their reasoning performance remains a vibrant field of inquiry. To address this, we present d2, a specialized reasoning framework designed for masked DLMs. The core innovation of our approach is a novel policy gradient algorithm that depends on precise calculations of sampling trajectory likelihoods.

Calculating these likelihoods directly is typically computationally prohibitive for masked DLMs. To overcome this, we have developed a suite of estimators customized for different model architectures. For DLMs compatible with "any-order decoding," we introduce d2-AnyOrder. This method delivers exact trajectory likelihood estimates in a single model pass. However, our empirical analysis of commonly used DLMs reveals that any-order decoding is not universally available in current implementations.

For standard masked diffusion models lacking this capability, we propose d2-StepMerge. This approach approximates trajectory likelihoods, offering a controllable trade-off between computational cost and approximation accuracy through an analytically tractable method. Our experiments demonstrate that d2 surpasses established reinforcement learning baselines when applied to popular DLMs. Furthermore, it establishes new state-of-the-art performance on logical reasoning benchmarks (Sudoku and Countdown) and mathematics reasoning datasets (GSM8K and MATH500).

The source code and a detailed blog post are available at: https://guanghanwang.com/d2

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC