Title: Analyzing Decoding in Order-Agnostic Language Models: Chain-Rule Deviation and Uniform Spreading

Abstract:

Order-agnostic language models (OALMs), such as discrete diffusion language models (dLLMs), are designed to predict masked tokens based on arbitrary conditioning sets. This capability enables the generation or scoring of sequences under any reveal order during inference. In this study, we present three key findings regarding LLaDA-2.1. First, we demonstrate that learned conditionals do not constitute exact factorizations of a single coherent joint distribution. Specifically, altering only the reveal order can shift the target log-likelihood by as much as 0.49 nats per token. Consequently, relying solely on likelihood conflates the inherent difficulty of the content with artifacts dependent on the decoding path. Second, while confidence-first (CF) decoding is technically order-agnostic, its reveal sequences for content tokens tend to align closely with left-to-right (L2R) ordering. Third, we introduce a complementary diagnostic tool centered on the shape of the confidence trace. A uniform-spreading theorem establishes that, given a fixed total likelihood, target recoverability is optimized when per-step confidence is distributed uniformly. This deviation leads us to propose $\mathrm{Var}(\log q_t)$ as a metric for evaluating decoding paths. Our evaluation across the C4 dataset and four downstream benchmarks reveals that low variance effectively distinguishes structured paths from random orderings, and that variance levels are consistently correlated with downstream accuracy. These findings advocate for the joint reporting of mean confidence and confidence variance when assessing decoding strategies in OALMs.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC