Title: HOLA: Holistic Multi-Modal Alignment for Open-Set 3D Recognition

Abstract

Open-set 3D recognition demands models capable of generalizing toward rare or previously unseen categories. While recent methods have attempted to bridge this gap by distilling language-vision knowledge into 3D encoders, they often depend on computationally intensive 2D Vision Transformers (ViTs). Furthermore, these approaches typically align each point cloud with just one image or caption, which anchors the resulting representations to limited perspectives. To overcome this limitation, we propose a strategy that aligns every point cloud with a diverse set of images and textual descriptions, thereby fostering a more comprehensive understanding of 3D objects.

Implementing this concept requires a novel loss function capable of simultaneously aligning a 3D instance with multiple corresponding signals—specifically, multi-view images and various text descriptions—while effectively distinguishing between positive aggregation and negative competition. We introduce the "decoupled multi-positive contrastive loss" to achieve this. This formulation strengthens the loss function’s ability to focus on difficult negatives in a hardness-aware manner, thereby mitigating the "spotlight crowding" effect that arises when numerous positives compete within the same softmax distribution against all negatives.

Additionally, we incorporate a lightweight text adapter that is applied exclusively to web captions. This mechanism narrows the domain gap between noisy web data and curated annotations, facilitating the effective utilization of large-scale unsupervised text. Our model achieves state-of-the-art open-vocabulary performance on long-tail benchmarks, delivering significant zero-shot accuracy gains while maintaining high frame rates.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC