Title: Optimizing Layer Attention Efficiency by Pruning Redundant Retrievals

Abstract:

Recent studies indicate that layer attention mechanisms, which facilitate deeper interaction within neural network architectures, have played a crucial role in advancing deep learning models. Nevertheless, current layer attention techniques are plagued by redundancy, as the attention weights generated by neighboring layers tend to converge toward high similarity. This duplication leads to multiple layers extracting largely identical features, which diminishes the model’s representational power and prolongs training durations. To mitigate this challenge, we present a new method for measuring redundancy by calculating the Kullback-Leibler (KL) divergence between adjacent layers. Furthermore, we propose an Enhanced Beta Quantile Mapping (EBQM) technique designed to precisely detect and bypass redundant layers, thus ensuring the stability of the model. The resulting Efficient Layer Attention (ELA) architecture not only boosts overall performance but also significantly improves training efficiency, cutting training time by 30% while delivering superior results in applications like image classification and object detection.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC