Title: Variational Learning for Insertion-based Generation

Non-monotonic sequence generation techniques, including masked diffusion models, offer a flexible alternative to traditional left-to-right autoregressive modeling by permitting tokens to be produced in non-fixed and prescribed sequences. While these methods offer practical benefits, most current non-monotonic models are order-agnostic and depend on a fixed-length grid. This reliance restricts their capacity to handle variable-length generation and adapt the insertion order.

This study presents a probabilistic framework designed to learn insertion order within variable-length insertion models. We establish a bijective correspondence between insertion trajectories and permutations, facilitating an exact reparameterization of the data likelihood as a summation over permutations. Leveraging this finding, we propose the Insertion Process (IP), a stochastic generative model that simultaneously determines where to insert, what content to insert, and when to conclude the process. The model is trained using permutation-based variational inference.

In contrast to previous fixed-canvas approaches, the IP model natively accommodates variable-length generation and acquires data-driven preferences regarding insertion orders. Evaluations conducted on molecular string generation and goal-conditioned planning reveal that learning the insertion order enhances both modeling quality and generalization in domains lacking a canonical left-to-right structure.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC