Title: Symbol-Invariant Transformer for Open-Vocabulary Learning: Why Names Are Irrelevant

Abstract: Current neural network architectures lack a robust framework for managing interchangeable tokens—symbols that carry identical semantic weight but are visually distinct, such as bound variables. Consequently, models relying on static vocabularies frequently fail to generalize to new symbols, even when the core meaning of the input remains consistent. To address this, we introduce a novel Transformer-based mechanism that is theoretically guaranteed to remain invariant to the renaming of these interchangeable tokens. Our method utilizes parallel embedding streams to isolate the specific influence of each interchangeable token within the input data, coupled with an aggregated attention mechanism that facilitates structured information exchange across these streams. Our experimental findings validate the theoretical assurances of our approach and highlight significant improvements in performance on open-vocabulary tasks that necessitate generalization to previously unseen symbols.

Project Page: https://bu-depend-lab.github.io/Symbol-Invariant-Transformer/

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC