Title: FedMTFI: Optimizing Multi-Teacher Knowledge Distillation via Feature Importance in Heterogeneous Federated Learning

Abstract:

Federated Learning (FL) offers a decentralized framework for collaborative model training, ensuring data privacy by allowing devices to exchange only model weights rather than sensitive raw information. This approach keeps personal data stored locally and secure. However, practical deployments often face challenges due to the non-uniform distribution of data across devices and significant disparities in computational resources and memory capacity. These heterogeneities complicate efforts to sustain consistent system-wide performance.

To overcome these limitations, this paper introduces FedMTFI, a novel architecture that integrates multi-teacher knowledge distillation (MTKD) with feature importance analysis to enhance FL performance in heterogeneous settings. In the FedMTFI framework, clients are grouped into clusters based on comparable hardware specifications and model architectures. Each cluster trains a dedicated model on its locally held, non-independent and identically distributed (non-IID) data. Within each cluster, individual clients refine their respective models using only their private, local datasets.

Subsequently, the central server aggregates the locally trained models from each cluster using the FedAvg algorithm to generate multiple prototype models. These prototypes then function as teacher models, guiding the training of a global, generalized student model through MTKD. A distinctive feature of FedMTFI is the incorporation of Shapley values (SHAP) to highlight critical features during the distillation process, thereby boosting both model accuracy and interpretability. Experimental evaluations demonstrate that FedMTFI surpasses traditional FL algorithms in accuracy and exhibits superior effectiveness when handling non-IID data conditions.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC