Title: CHASMBrain: A Coarse-to-Fine Hierarchical Framework Using Sequential Mamba for Brain Reconstruction

Abstract:

Bridging the gap between deep visual representations and the human visual system remains a pivotal challenge in computational neuroscience. Although contemporary vision models excel at image recognition, it is still unclear how well their internal structures mirror the hierarchical organization of the human visual cortex. To address this, we introduce CHASMBrain, a novel two-stage hierarchical framework designed for encoding images into fMRI data. Inspired by the functional architecture of the visual cortex, our approach utilizes a dual-stream Mamba design that distinctly separates and processes global semantic tokens from local spatial patches.

The framework operates on a coarse-to-fine principle. In the first stage, the model predicts denoised activations at the region-of-interest (ROI) level. The second stage then refines these initial predictions into precise, full voxel-level outputs by employing a Mamba-VAE. We validated our method using the Natural Scenes Dataset (NSD), where it achieved a Pearson correlation of 0.429 and a Mean Squared Error (MSE) of 0.261. These results surpass all tested baselines, including ridge regression and linear probes based on DINOv2.

Beyond standard predictive metrics, causal branch-ablation studies uncovered an asymmetric specialization within the model: the patch stream is causally linked to early visual cortex regions (retinotopic areas), whereas the CLS stream provides broader semantic context to higher-order brain regions. This relationship is causal rather than merely correlational. Furthermore, cross-subject transfer tests indicate that the learned backbone generalizes effectively across different individuals with only minimal per-subject adaptation, implying that the model successfully captures a shared, subject-agnostic visual representation.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC