Title: Assessing Biological Relevance in Foundation Models: An Evaluation of Attention Coherence via Spatial Transcriptomics in Glioblastoma

Abstract:

It remains unclear whether the attention mechanisms employed by pathology foundation models reflect authentic biological processes, a gap that poses significant challenges for clinical adoption and regulatory clearance. To address this, we introduce a spatial transcriptomics-driven framework designed for hypothesis-free, orthogonal assessment of model attention. This approach was applied to five prominent pathology foundation models—CONCH v1.5, UNI v2, Virchow2, GigaPath, and H-Optimus-1—alongside a ResNet50 baseline.

By leveraging attention-based multiple instance learning, we developed single-task and multi-task predictive models for five specific molecular alterations in glioblastoma. These models were trained on the CPTAC cohort and externally validated using an independent TCGA dataset. We further assessed the biological fidelity of the attention maps by comparing them against 87 transcriptional signatures, utilizing co-registered Visium spatial transcriptomics data derived from 18 glioblastoma samples.

Our internal analysis revealed that no single encoder consistently outperformed the others across all prediction tasks. Furthermore, performance rankings shifted significantly during external validation. The attention maps exhibited a distinct gradient of enrichment, showing a five-fold difference between pathway-level associations (Cohen's d=0.329) and individual gene associations (d=0.055). This pattern suggests that the models capture emergent, multi-gene transcriptional programs rather than isolated molecular events. Notably, spatial smoothness in attention maps did not guarantee biological coherence, and various encoders focused on different biological compartments. Ultimately, this framework offers an objective, quantitative method for evaluating what foundation models derive from histopathology, advancing the field beyond subjective reviews of saliency maps.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC