Hyperbolic and Evidence-Prioritized Experts for Large Vision-Language Models
Title: Leveraging Hyperbolic Geometry and Evidence-First Prioritization in Large Vision-Language Models
Abstract:
While Large Vision-Language Models (LVLMs) have achieved remarkable success in multimodal applications through extensive training and scaled architectures, recent efforts to enhance computational efficiency have integrated Mixture of Experts (MoE) mechanisms. However, conventional MoE implementations often apply symmetric architectures to both visual and linguistic inputs, failing to account for the fundamental processing asymmetry between these modalities. This oversight leads to two significant challenges. First, the relationship between text and vision is hierarchical rather than parallel; textual queries usually describe specific facets of a broader visual context. Standard Euclidean expert spaces are ill-equipped to represent such containment-based structures. Second, as language processing moves into deeper network layers, experts tend to rely increasingly on parametric memory rather than the immediate visual and linguistic evidence, resulting in a loss of grounding.
To overcome these limitations, we introduce AsyMoE, a novel architecture designed to explicitly model this inherent asymmetry via three distinct expert categories. Intra-modality experts are dedicated to processing information unique to each specific modality. Hyperbolic inter-modality experts utilize negative curvature geometry to effectively capture the hierarchical nature of cross-modal relationships. Furthermore, evidence-priority language experts are engineered to minimize reliance on parametric memory, thereby preserving contextual grounding across all network depths. Our extensive experimental results show that AsyMoE consistently outperforms baseline approaches, delivering an average improvement of 1.5% over existing MoE variants and up to 3.8% on tasks particularly sensitive to hallucinations. Additionally, the model reduces parameter activation by 25.45% compared to dense models.
Source: arXiv Generated at: 2026-06-02 00:00:00 UTC
