Title: Evaluating Sign Language Models Through Linguistic Analysis Using Minimal Translation Pairs

Abstract: Sign language AI has traditionally trailed behind spoken language counterparts in both text and speech modalities. Although recent advancements have significantly boosted performance on tasks such as isolated sign recognition and sign language translation, it remains uncertain how effectively current models grasp diverse linguistic structures within sign language. Furthermore, the degree to which these systems leverage cues from the multiple articulators involved in signing—such as the hands, face, and upper body—is not well understood. To address this gap, we present ASL-MTP (American Sign Language Minimal Translation Pairs), a novel benchmark dataset designed for linguistic analysis. This dataset categorizes various sign language phenomena into corresponding minimal translation pairs. As a case study, we apply ASL-MTP to examine a leading state-of-the-art model for ASL-to-English translation. By ablating specific input cues during both training and inference phases, we conduct a targeted evaluation of the model’s performance across the phenomena defined in ASL-MTP. Our findings indicate that while the model exceeds chance levels on most phenomena, it exhibits a heavy reliance on manual cues and frequently fails to incorporate essential non-manual signals.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC