Title: On the Expressive Power of Permutation-Equivariant Weight-Space Networks

Abstract

Weight-space learning involves neural architectures that function directly on the parameters of other neural networks. Driven by the increasing accessibility of pretrained models, recent studies have highlighted the efficacy of weight-space networks across numerous applications. To enhance generalization, state-of-the-art weight-space networks typically employ permutation-equivariant designs. However, such constraints may compromise expressive power, necessitating a theoretical examination. This analysis is particularly complex because, unlike other structured domains, weight-space learning focuses on mappings that operate across both weight and function spaces. Although previous research has offered some partial insights into expressivity, a complete characterization remains absent. This paper fills this void by introducing a systematic theory regarding the expressivity of weight-space networks. We demonstrate that all major permutation-equivariant networks share equivalent expressive capabilities. Furthermore, we prove universality in both weight- and function-space contexts under mild, natural assumptions regarding input weights, while also identifying the edge-case conditions under which universality fails. Leveraging these theoretical findings, we reveal that minor adjustments to current weight-space models can achieve a 34% performance gain over previous state-of-the-art results, underscoring the practical value of our framework.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC