Title: Why Sequence Counts in Agentic Memory: Implementing Segment Trees for Long-Horizon Agents

Abstract:

Extended interaction cycles require conversational agents to navigate shifting objectives, tasks, and events. While these interactions are inherently time-based, current memory frameworks often prioritize thematic relevance, potentially overlooking the chronological sequence of occurrences. To address this, we present Segment Tree Memory (SegTreeMem), a novel architecture that structures conversation history as a temporally sorted Segment Tree comprising individual utterances. This system utilizes an online rightmost-frontier update mechanism to incrementally add new utterances, thereby maintaining chronological integrity while establishing hierarchical memory blocks. During the retrieval phase, SegTreeMem distributes relevance scores across the tree structure, effectively merging local semantic alignment with hierarchical temporal context. Our evaluations across two large language model backbones and three long-horizon memory benchmarks demonstrate that SegTreeMem outperforms flat retrieval, graph-based, and standard tree-based memory baselines in terms of response accuracy. Furthermore, our analysis of temporal-order permutations confirms that these performance enhancements are contingent upon preserving chronological order during memory formation, reinforcing the hypothesis that temporal sequencing is a fundamental structural element for agentic memory.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC