Title: Knowledge Editing in Masked Diffusion Language Models

Abstract:

Knowledge editing focuses on modifying or correcting factual information stored within language models. The prevalent "locate-then-edit" strategy accomplishes this through a two-phase process: first identifying the specific location of a fact within the model, and subsequently adjusting the weights at that site. Until now, these techniques have been applied exclusively to autoregressive models (ARMs). It remains unclear whether the foundational assumptions of these methods are valid for masked diffusion models (MDMs), which generate text via iterative denoising and model context bidirectionally, rather than relying on next-token prediction.

This study investigates this gap by adapting the locate-then-edit framework to MDMs. We conducted a comparative analysis between two MDMs (LLaDA and Dream) and two ARMs (LLaMA and Qwen), ensuring the models were of comparable scale. Our results yield two primary insights. First, the optimal location for applying edits appears to be consistent across different model paradigms. Causal tracing reveals that, in both MDMs and ARMs, the most effective edits target the same early-to-mid-layer MLPs located at the final subject token. However, this convergence in location does not ensure convergence in results.

While single-token edits are successful in both model types, performance in MDMs deteriorates systematically as the length of the target text increases, a trend not observed in ARMs. We attribute this decline to the mechanism of generation: creating multi-token targets requires the model to pass through partially unmasked intermediate states, which were not accounted for during the initial optimization of the edit. To address this, we propose a straightforward correction method that optimizes the edit specifically for these intermediate states. This adjustment significantly restores the model’s ability to handle multi-token edits.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC