Title: Unified Semantic Transformer for 3D Scene Understanding

Abstract:

Achieving a comprehensive understanding of 3D environments requires the ability to capture and interpret unstructured spatial data. However, the intricate nature of real-world settings has led to the development of models that are largely specialized for individual tasks. To address this limitation, we present UNITE, a Unified Semantic Transformer designed for 3D scene understanding. This novel feed-forward neural network consolidates a wide array of 3D dense semantic indoor tasks into a single, cohesive framework.

UNITE functions in a fully end-to-end manner on previously unseen scenes, delivering full 3D semantic geometry inference in just a few seconds. By utilizing only RGB images as input, the model directly predicts multiple dense semantic attributes, such as 3D scene segmentation, instance embeddings, open-vocabulary features, and articulations. The training process combines 2D distillation with a strong emphasis on self-supervision, augmented by innovative multi-view losses that enforce consistency across different 3D perspectives.

Our results indicate that UNITE delivers state-of-the-art performance across various dense indoor semantic benchmarks. Notably, it frequently outperforms models specifically designed for individual tasks and even surpasses methods that rely on ground truth 3D geometry. For more details, visit the project website at unite-page.github.io.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC