Title: Reevaluating Positional Supervision Strategies in Masked Diffusion Language Model Training

Abstract:

Masked diffusion language models (MDLMs) have recently gained traction as viable alternatives to autoregressive models by generating text through the parallel unmasking of tokens. Functionally, these models operate as parallel decoders optimized using position-wise cross-entropy (CE) loss, mirroring the architecture and training methodology of non-autoregressive translation (NAT). In the context of NAT, it has been established that parallel decoders trained with CE are highly susceptible to minor positional shifts, as the loss function imposes severe penalties for such deviations. This study investigates whether MDLMs exhibit similar sensitivity to positional shifts during iterative decoding.

To test this hypothesis, we implemented a controlled intervention during the decoding process. Our experiments with LLaDA-8B-Instruct on the Arena-Hard benchmark revealed that shifting merely 1% of the generated tokens by a single position significantly diminishes the model’s win rate against the unintervened baseline. These findings confirm that MDLMs are indeed sensitive to small positional displacements under iterative parallel decoding.

Addressing this vulnerability, we adapted connectionist temporal classification (CTC)—an alignment-flexible objective previously shown to mitigate such sensitivity in NAT—for the supervised fine-tuning of MDLMs. By relaxing the rigid position-wise matching requirement inherent to CE, CTC allows the loss function to accommodate minor positional variations. Specifically, we refined the CTC objective by introducing a special token to handle positional uncertainty between target tokens and output positions, alongside an updated collapse map designed to preserve the surface forms of targets.

Evaluations across four open-ended generation benchmarks demonstrate that this approach yields consistent improvements over both the original model and a baseline trained with matched cross-entropy. These gains were statistically significant across all four benchmarks. Consequently, our results highlight training-time alignment flexibility as a critical design parameter for MDLM supervised fine-tuning, offering a complementary perspective to the inference-time strategies examined in previous research.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC