Platonic Transformers: A Robust Solution for Equivariance

Abstract:

Transformers are ubiquitous in modern machine learning but typically lack the inductive biases necessary to handle the geometric symmetries prevalent in scientific and computer vision applications. Current approaches to achieving equivariance often compromise the efficiency and adaptability that define Transformer architectures, relying instead on complex and computationally expensive mechanisms. To address this limitation, we propose the Platonic Transformer, a model designed to bridge this gap.

Our approach establishes attention mechanisms relative to reference frames derived from Platonic solid symmetry groups. This methodology generates a structured weight-sharing protocol that ensures equivariance to both continuous translations and Platonic symmetries. Crucially, this is achieved without altering the standard Transformer architecture or increasing its computational overhead. We demonstrate that this specific form of attention is formally equivalent to a dynamic group convolution. This equivalence highlights the model’s ability to learn adaptive geometric filters and facilitates the development of a highly scalable, linear-time convolutional variant.

Empirical evaluations across multiple domains confirm the efficacy of the Platonic Transformer. The model delivers competitive results on benchmarks spanning computer vision (CIFAR-10), 3D point cloud analysis (ScanObjectNN), and molecular property prediction (QM9 and OMol25). These outcomes underscore the advantage of integrating geometric constraints into Transformer designs without incurring additional computational costs.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC