Title: Inference-Time Scaling Strategies Can Enhance Generative Pre-Training Methodologies

Abstract: Current approaches to generative pre-training frequently rely on a misleading binary classification that separates autoregressive models, designed for discrete signals, from diffusion models, intended for continuous signals. We contend that this distinction is erroneous, as it mistakenly conflates the model architecture, data representation, training objective, and the inference process. Specifically, autoregression functions as an inference mechanism that builds sequences via normalized conditional sampling, whereas diffusion operates as a refinement process that iteratively updates an existing state. Consequently, the more relevant comparison lies not between autoregressive and diffusion paradigms, but rather between discrete tokens trained with cross-entropy and continuous tokens trained using diffusion-style objectives, alongside the respective sampling algorithms employed. From this viewpoint, advancements in algorithms should focus on improving inference-time efficiency across two primary dimensions: sequence expansion and state refinement. We propose that the inference procedure should be designed prior to the training objective, as a training method cannot rectify an inference map that lacks essential arguments or enforces an inaccurate factorization. We demonstrate this principle by examining the target-time constraints inherent in DDIM-style samplers, the joint-distribution limitations found in multi-token prediction, and recent innovations in flow-map and few-step distillation techniques that explicitly parameterize long-range inference moves.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC