Title: Enhancing Transformers with Dynamic Short Convolutions

Original: arXiv:2606.03825v1 Announcement Type: New Research

Abstract: Transformers have established themselves as the preeminent architecture for large language models, a status largely attributed to the scalability and adaptability provided by attention mechanisms, feed-forward networks, residual connections, and normalization techniques. This study presents dynamic short convolutions as a novel neural network primitive designed to bolster Transformer performance. In contrast to static short convolutions, dynamic convolutions generate filters that vary based on input data, thereby maintaining the inherent locality bias of convolutional operations while significantly enhancing model expressivity. Preliminary experiments demonstrate that integrating dynamic short convolutions into key, query, and value representations yields superior results on difficult associative recall tasks when compared to static convolutional alternatives. In language modeling trials involving models with parameter counts between 150 million and 2 billion, dynamic convolutions consistently surpassed both standard Transformers and those enhanced with static short convolutions. Scaling law analysis reveals a 1.33$\times$ compute efficiency gain over compute-equivalent Transformers when dynamic convolutions are applied to key, query, and value vectors, and a 1.60$\times$ advantage when these convolutions are inserted after every linear layer. Furthermore, dynamic convolutions provide performance benefits for linear RNNs, such as Mamba-2 and Gated DeltaNet, as well as mixture-of-experts frameworks. To facilitate practical adoption, we have developed custom Triton kernels that allow for efficient training with only a moderate increase in end-to-end latency. These findings indicate that dynamic short convolutions serve as a scalable, hardware-efficient, and highly expressive component for the advancement of Transformer-based language models.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC