Title: Establishing a Unified Geometric Framework for Topological Alignment Between Transformer Architectures and Human Brain Networks

Abstract:

Existing research on aligning artificial intelligence with the brain is often restricted by particular inputs and tasks, which hinders the ability to capture organizational characteristics across models with varying modalities. To address this, our study concentrates on Transformer-based systems and proposes a novel topological alignment space connecting brain and model structures. Instead of deriving alignment from neural mechanisms, we investigate it via graph-based organizational features, specifically mapping the intrinsic spatial attention topology of these models onto canonical human intrinsic connectivity networks (ICNs). This approach facilitates a comparison that is both modality-agnostic and task-independent, spanning vision, language, and multimodal systems at the level of organizational structure.

By analyzing 151 Transformer-based models across different modalities and scales, we identified a continuous arc-shaped distribution indicative of varying topological alignment degrees. Aligning with their respective training goals, models designed for global semantic abstraction demonstrated stronger associations with higher-order ICNs, whereas those focused on local details correlated more closely with lower-level ICNs.

Unexpectedly, we also detected several counterintuitive trends. DINOv2 showed reduced alignment compared to earlier versions, and distilled DeiT models exhibited a scaling inversion where larger models aligned less effectively with higher-order ICNs. Additionally, both fine-tuning and instruction tuning appeared to have minimal impact on alignment scores. Notably, topological alignment scores did not significantly correlate with ImageNet-1K Top-1 accuracy among 30 vision Transformers (r=0.266, p=0.156). This research offers a new quantitative framework for evaluating the organizational properties of Transformer-based models through brain-referenced topological mapping.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC