Title: IDLM: Inverse-distilled Diffusion Language Models

Original: arXiv:2602.19066v2 Announce Type: replace-cross

Abstract: Although Diffusion Language Models (DLMs) have demonstrated impressive capabilities in text generation, their practical application is hindered by slow inference speeds caused by multi-step sampling processes. To mitigate this bottleneck, we adapt Inverse Distillation—a method originally designed to speed up continuous diffusion models—to the discrete domain. This adaptation, however, presents significant theoretical and practical hurdles. Theoretically, the inverse distillation objective does not inherently guarantee uniqueness, raising the risk of converging to suboptimal solutions. Practically, performing backpropagation within discrete spaces is complex and prone to instability.

We address these issues by first establishing a theoretical proof that our inverse formulation yields a unique solution, which ensures robust optimization. Furthermore, we implement gradient-stable relaxations to facilitate effective model training. Empirical evaluations across various DLMs confirm that our proposed approach, Inverse-distilled Diffusion Language Models (IDLM), accelerates inference by reducing the required steps by a factor of 4 to 64, all while maintaining the high-quality text generation of the original teacher model. Researchers and practitioners can access the project code, model checkpoints, and video tutorials at: https://david-cripto.github.io/idlm-project-page

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC