Title: EndPrompt: Efficient Long-Context Extension via Terminal Anchoring

Abstract:

Adapting large language models to handle extended context windows traditionally demands training on sequences at the desired length, a process that incurs prohibitive quadratic memory and computational expenses, rendering long-context adaptation costly and hard to replicate. To address this, we introduce EndPrompt, a novel approach that enables effective context expansion using exclusively short training sequences. Our central hypothesis is that it is unnecessary to construct full-length inputs to expose a model to long-range relative positional distances. Instead, we maintain the original short context as a cohesive initial segment and append a concise terminal prompt as a secondary segment, assigning it positional indices close to the target context length. This two-part structure embeds both local and long-range relative distances within a compact physical sequence, thereby preserving the semantic continuity of the training text—a critical advantage over chunk-based simulation methods that fragment contiguous contexts.

Through theoretical analysis based on Rotary Position Embedding and the Bernstein inequality, we demonstrate that position interpolation imposes a strict smoothness constraint on the attention function, while shared Transformer parameters further mitigate unstable extrapolation to unobserved intermediate distances. When applied to LLaMA-family models to extend their context window from 8K to 64K, EndPrompt attained an average RULER score of 76.03 and achieved the highest average performance on LongBench. It outperformed LCEG (72.24), LongLoRA (72.95), and full-length fine-tuning (69.23), all while requiring significantly fewer computational resources. These findings suggest that long-context generalization can be fostered through sparse positional supervision, thereby challenging the dominant belief that dense training on long sequences is essential for reliable context-window extension. The code is available at https://github.com/clx1415926/EndPrompt.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC