Title: SWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentation

Abstract:

While large-scale vision foundation models have significantly advanced dense prediction tasks like semantic segmentation, their substantial computational footprint renders them impractical for resource-limited environments. This limitation has spurred interest in knowledge distillation as a method to transfer the capabilities of these massive models to lightweight student networks. However, a fundamental architectural mismatch exists: modern teacher models are primarily transformer-based, leveraging global context, whereas efficient student networks are typically convolutional, characterized by locally biased receptive fields. Current distillation techniques often presuppose architectural similarity and depend on direct feature mimicry. Consequently, they fail to bridge this representational divide, overlooking the structured spatial dependencies and discriminative organization essential for precise semantic segmentation.

To address these challenges, we introduce SWARD, a novel knowledge distillation framework built on two complementary mechanisms. First, we propose the Multi-Scale Windowed Attention Distillation (MWAD) module. This component aligns attention-based relationships between the teacher and student within window partitions that are stochastically shifted, with offsets randomly resampled at each training iteration. This approach eliminates bias associated with fixed window boundaries. Coupled with its multi-scale design, MWAD effectively captures both short- and long-range spatial dependencies. Second, we present Prototype Discriminative Regularization (PDR), a specialized loss function designed to refine the student’s feature distribution. By enforcing intra-class compactness and inter-class separation, PDR sharpens the discriminative structure of the student model, achieving results superior to what feature mimicry alone can provide given the student’s reduced capacity. Experimental evaluations across diverse vision applications, including urban scene parsing and medical image segmentation, demonstrate that SWARD delivers state-of-the-art performance.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC