Title: The Intersection of Induction and Biology: How Protein Language Models Identify Repeats

Abstract:

Protein sequences frequently contain recurring segments, appearing either as precise duplicates or as approximate variations marked by mutations. Given the critical role these repeats play in determining protein structure and function, researchers have spent decades developing algorithms to identify them. Recent studies have demonstrated that protein language models (PLMs) are capable of recognizing these repeats, a capability evidenced by their performance in masked-token prediction tasks. This study delves into the internal mechanics of PLMs to understand how they detect both exact and approximate repetitions. We discover that the mechanism employed for approximate repeats effectively encompasses the process used for exact ones. By characterizing this unified mechanism, we identify two primary phases: initially, PLMs construct feature representations through a combination of general positional attention heads and biologically specialized elements, such as neurons tuned to encode amino acid similarity. Subsequently, induction heads focus on tokens that align across different repeated segments, thereby facilitating the identification of the correct answer. These findings illustrate how PLMs address this biological challenge by integrating language-based pattern recognition with domain-specific biological insights, laying the groundwork for future investigations into more complex evolutionary dynamics within these models.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC