Title: Learning Unmasking Sequences in Masked Diffusion Models

Masked diffusion models have achieved significant success in modeling data distributions for discrete sequences, particularly in fields like text generation and protein design. Typically, these models synthesize data by progressively revealing tokens from an entirely masked starting point, relying on either random selection or heuristics derived from denoiser probabilities to determine the unmasking sequence.

This paper introduces a novel method for learning the unmasking order by integrating a lightweight policy network with a standard diffusion model. The proposed approach utilizes a loss function that reweights components of the masked diffusion loss based on policy probabilities, thereby encouraging the model to prioritize positions where the denoiser is most likely to be accurate.

We evaluate this weighted loss framework in two distinct scenarios: first, by training only the policy network while keeping a pre-trained denoiser fixed; and second, by jointly training both the policy and the denoiser to facilitate mutual adaptation. Our results indicate that this learned strategy surpasses traditional heuristics in tasks highly sensitive to token sequence, including combinatorial optimization and protein structure prediction.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC