Title: LongLive-RAG: A Universal Retrieval-Augmented Framework for Extended Video Synthesis

Abstract

While autoregressive (AR) video diffusion models facilitate the synthesis of variable-length content, they frequently struggle with long-horizon generation due to identity drift and the compounding nature of accumulated errors. To optimize efficiency, current approaches typically employ sliding-window attention. However, this method establishes an irreversible generation path; once the active window incurs visual inaccuracies, subsequent frames are forced to condition on this flawed trajectory, causing the output to drift further from the intended content.

We propose a solution by reframing long video generation as a Retrieval-Augmented Generation (RAG) challenge. Instead of depending exclusively on the immediate recent window, our approach treats previously generated latents as a dynamic, searchable repository. We introduce LongLive-RAG, a versatile retrieval framework designed for AR video generation. During the creation of each new block, the system utilizes a query embedding to locate pertinent historical latents. This lightweight retrieval process introduces minimal computational overhead compared to the generation task itself, enabling the generator to leverage non-local context rather than being confined to the immediate window.

To enhance the discriminative power of this retrieval mechanism, we present the Window Temporal Delta Loss. This loss function mitigates redundant local similarities and incentivizes embeddings to encode significant temporal changes. Collectively, these innovations help mitigate the error accumulation inherent in sliding-window attention. Empirical evaluations across various AR backbones and generation durations demonstrate enhanced quality in long videos, achieving the highest average rank on VBench-Long. To the best of our knowledge, LongLive-RAG represents the first open-ended AR long video generation method to conceptualize self-generated latent history as a content-addressable retrieval memory. The source code is accessible at https://github.com/qixinhu11/LongLive-RAG.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC