Title: CR-JEPA: A Cross-Modal Joint-Embedding Predictive Framework for Remote Sensing Image Retrieval

Abstract:

Retrieving semantically linked scenes across diverse sensing modalities is a core objective of cross-modal remote sensing image retrieval. This task presents significant hurdles, as paired observations often exhibit marked discrepancies in imaging physics, spatial resolution, spectral setups, and visual characteristics. Additionally, relying on a single retrieval projection optimized for one objective may fail to simultaneously achieve cross-modal semantic alignment and preserve neighborhood structures within the same modality.

To address these challenges, we introduce CR-JEPA, a Cross-modal Retrieval Joint-Embedding Predictive Architecture designed for dual-modality remote sensing retrieval. The proposed model integrates modality-specific stems with a shared transformer trunk, utilizing JEPA-style predictive mechanisms to estimate masked latent target features both within and across different modalities. Drawing inspiration from LeJEPA, we implement Sketched Isotropic Gaussian Regularization on raw retrieval projections. This technique helps stabilize the embeddings and prevents model collapse. Furthermore, CR-JEPA utilizes a decoupled-head architecture, featuring a unified head dedicated to same-modal retrieval and a separate head optimized for cross-modal search.

We assessed the performance of CR-JEPA using the BEN-14K, CBRSIR_VS, and DSRSID datasets. Experimental results on BEN-14K demonstrate that CR-JEPA significantly outperforms X-JEPA, boosting S1-to-S2 retrieval accuracy from 61.23% to 75.82% and S2-to-S1 retrieval from 63.73% to 75.40%. In addition to these improvements in cross-modal performance, CR-JEPA delivers competitive results in same-modal retrieval while requiring fewer parameters.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC