Title: When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models

Abstract

Graph Language Models (GLMs) represent a burgeoning strategy for tailoring Large Language Models (LLMs) to tasks involving graph data. This approach works by encoding graph topology and node attributes into "graph tokens," thereby enabling LLMs to handle structured graph inputs alongside textual commands in a unified manner. However, the internal mechanisms by which LLMs interpret these tokens remain opaque, raising questions about whether graph tokens effectively serve as carriers of structural information. This study investigates the processing of graph data by LLMs through an examination of graph-token dynamics within prominent GLM frameworks.

Key Findings

Our analysis reveals a critical distinction between the internal saliency of graph tokens and their actual utility in utilizing graph information. We observe that "graph sink tokens" consistently appear as outliers at the activation level, characterized by substantial activation values across a limited number of hidden-state dimensions and a tendency to occupy early positions in the graph-token sequence. Despite this prominence, such activation-level saliency does not correlate with being the primary conduit for graph data. Contrary to the behavior of classical attention sinks seen in language and vision-language models, graph sink tokens do not inherently draw the highest attention weights from query tokens. Furthermore, interventions involving the pruning, repositioning, and swapping of these tokens demonstrate that graph sink tokens are not the most critical semantic or structural elements for downstream prediction tasks.

Implications

These findings indicate that the current mapping of graph structures into LLM token spaces does not yield a fully functional, topology-aware internal representation. Instead, there is a notable decoupling between activation-level prominence and graph-semantic utility. This disconnect highlights inherent limitations in the existing mechanisms governing graph-token construction, placement, and alignment within GLMs.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC