Title: VISReg: Applying Variance-Invariance-Sketching Regularization to JEPA Training

Abstract: Self-supervised learning techniques typically avert embedding collapse through the use of modeling heuristics or by explicitly regularizing the embedding space. Within the category of explicit regularization, VICReg enhances flexibility and interpretability by separating the regularization process into distinct variance and covariance objectives. However, because covariance only accounts for second-order statistics, it promotes decorrelation but fails to guarantee the complete distributional geometry required for stable training. While sketching-based approaches like SIGReg mitigate this issue by aligning embeddings toward an isotropic Gaussian, they often struggle with vanishing gradients during collapse and offer limited flexibility.

To address these limitations, we introduce Variance-Invariance-Sketching Regularization (VISReg). This method substitutes the standard covariance term with a sketching objective grounded in the Sliced-Wasserstein distance, which ensures the preservation of the full distributional shape. Simultaneously, it maintains a variance component to manage scale. By separating scale control from shape enforcement, VISReg merges the adaptability of VICReg with the rigorous distributional constraints of sketching methods, thereby delivering robust gradient signals even when collapse occurs.

Our experiments demonstrate that VISReg exhibits linear scalability and surpasses current regularization techniques on datasets of lower quality. Furthermore, it proves resilient in scenarios involving long-tailed distributions and low-rank structures. In evaluations on out-of-distribution datasets, VISReg pretrained on ImageNet-1K sets a new state-of-the-art. Additionally, when pretrained on ImageNet-22K, it achieves out-of-distribution performance comparable to DINOv2, despite DINOv2 being trained on ten times more data (LVD-142M).

Project page and code: https://haiyuwu.github.io/visreg

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC