Title: Language Modeling with Hyperspherical Flows

Abstract:

Discrete diffusion models have emerged as a compelling alternative to autoregressive (AR) systems, largely due to their capacity for parallel text generation. However, to maintain computational tractability, these models typically sample from factorized distributions, a constraint that reduces their expressive power compared to AR approaches. To address this, recent Flow Language Models (FLMs) employ continuous flows to transport noise to data via deterministic ordinary differential equations (ODEs), thereby bypassing the limitations of factorized sampling. Despite this advantage, standard FLMs operate on one-hot vectors, the dimensionality of which grows with vocabulary size, resulting in significant training costs. Furthermore, because all distinct one-hot embeddings are equidistant in $\ell_2$ space, introducing Gaussian noise lacks a meaningful semantic interpretation—unlike in image processing, where such noise gradually degrades structural integrity.

To overcome these challenges, we introduce $\mathbb{S}$-FLM, a latent FLM operating within a hypersphere. This approach generates sequences by rotating vectors along a velocity field learned via cross-entropy, effectively eliminating the computational burden of materializing one-hot vectors. While existing FLMs achieve Generative Perplexity (Gen. PPL) comparable to AR models, high-likelihood samples do not always yield correct results in verifiable domains like code and mathematics. $\mathbb{S}$-FLM significantly enhances the performance of continuous flow language models in large-vocabulary reasoning tasks. It narrows the performance gap with masked diffusion models under standard-temperature sampling ($T=1$), although a disparity persists under optimized low-temperature decoding ($T=0.1$).

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC