Title: Reasmory: Leveraging 3D Reconstruction as Explicit Memory for Enhanced Spatial Reasoning in Vision-Language Models

Abstract:

While Vision-Language Models (VLMs) are demonstrating nascent abilities in spatial reasoning, they frequently struggle with tasks demanding high precision, such as estimating distances, comparing directions, and reasoning from different viewpoints. In scenarios involving monocular videos or multi-view imagery, spatial information is often fragmented across numerous redundant observations, creating significant challenges for effective aggregation and utilization. Although Reconstruction-based Vision Foundation Models (VFMs) can naturally consolidate these observations into explicit spatial representations like point clouds, relying on them as open-ended tools proves fragile. VLMs often make erroneous tool calls, overlook necessary spatial transformations, or misinterpret intermediate outputs.

To address these limitations, we introduce Reasmory, a framework that treats spatial reasoning as the execution of structured programs over reconstructed spatial memory. Reasmory builds explicit 3D memory structures enriched with semantically anchored 3D object instances. It employs a lightweight Domain-Specific Language (DSL) to strictly govern how VLMs interact with this memory—specifically dictating how objects and cameras are queried, how viewpoints are transformed, and how observations are rendered. By parsing and validating generated programs prior to execution, Reasmory ensures more robust interactions with spatial data compared to unrestricted tool usage. Our experiments on spatial reasoning benchmarks using multi-view images and videos demonstrate performance improvements of 6% to 18% over powerful baseline models, including GPT-5-mini and Gemini-3-flash. These results highlight that explicit 3D memory yields the best outcomes when accessed through constrained, validated operations rather than free-form tool calls.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC