Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation
Title: Closing the 2D-3D Divide: A Hierarchical Semantic-Geometric Map for Vision-Language Navigation
Abstract:
Vision-Language Navigation (VLN) empowers embodied agents to navigate through unfamiliar environments by adhering to linguistic commands. Although recent advancements in vision-language models (VLMs) have been significant, a substantial semantic-geometric chasm persists. VLMs are highly proficient in processing language and interpreting 2D imagery, yet they frequently falter when it comes to 3D spatial reasoning. Furthermore, they often fail to grasp the causal link between specific actions and resulting spatial changes, leading to inconsistent navigation performance, especially in zero-shot scenarios.
To address this limitation, we introduce the Hierarchical Semantic-Geometric Map (HSGM). This framework converts 3D geometric data into a structured format that aligns with VLM capabilities, thereby creating a robust connection between the models and the physical environment. The HSGM is structured as a multi-channel, top-down map comprising three distinct layers:
- Geometric Level: Captures navigable spaces and identifies obstacles.
- Semantic Level: Depicts objects and their interrelations.
- Decision Level: Facilitates high-level task reasoning and goal selection.
In this architecture, the VLM functions as a high-level semantic planner. It interprets the spatial layout embedded within the HSGM to identify waypoints that are geometrically viable. Meanwhile, low-level, collision-free movement between these waypoints is managed by traditional path-planning algorithms. This design fully separates semantic reasoning from the mechanics of action execution. Additionally, the system breaks down intricate instructions into manageable subtasks to mitigate issues such as progress forgetting or hallucination during long-horizon navigation.
Extensive evaluations on the R2R-CE and RxR-CE benchmarks reveal that our zero-shot framework delivers state-of-the-art results, surpassing several supervised approaches. The source code can be accessed at https://github.com/Teacher-Tom/HSGM_public.
Source: arXiv Generated at: 2026-06-02 00:00:00 UTC
