Title: SAMatcher: Enhancing Robust Feature Matching via Co-Visibility Modeling with Segment Anything

Abstract:

Establishing reliable correspondences is a cornerstone of image processing, serving as the backbone for critical applications like Structure from Motion, visual localization, and image registration. While recent learning-based approaches have markedly advanced local feature representation, they predominantly function at the pixel or patch level, often neglecting explicit modeling of regions that are simultaneously visible across different views. To address this limitation, we introduce SAMatcher, a novel framework that redefines correspondence estimation through the lens of co-visibility modeling. Rather than relying on direct local feature matching, SAMatcher initially generates structured priors for correspondence by predicting co-visible region masks and bounding boxes. Leveraging the Segment Anything Model (SAM), the framework incorporates a symmetric cross-view interaction mechanism designed to facilitate bidirectional feature exchange and achieve semantic alignment across views. Additionally, we propose a comprehensive supervision strategy that simultaneously optimizes mask prediction and box localization, enforced by mask-box consistency constraints alongside mask learning and box regression. Extensive evaluations on demanding benchmarks reveal significant performance gains compared to current matching pipelines, especially in scenarios involving substantial viewpoint and scale changes. Our findings indicate that foundation models, originally crafted for monocular segmentation, can be successfully adapted for multi-view correspondence reasoning via explicit co-visibility modeling, thereby providing a fresh perspective on structured representation learning for image matching.

Code and project page: https://xupan.top/Projects/samatcher

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC