Title: The Unsampled Truth: Psychometrics in SLMs Measure Prompt Artifacts, Not Psychological Constructs

Abstract: Researchers typically operate under the assumption that when small language models (SLMs) are prompted for psychometric evaluations, their responses capture genuine semantic reasoning. To test this hypothesis, we examined 13 open-source models ranging from 0.6B to 14B parameters, employing a framework designed to disentangle semantic signals from prompt-induced artifacts. Through the systematic manipulation of personas, instructions, question items, and option symbols, we observed that variance caused by artifacts often supersedes the underlying semantic content. Consequently, in these instances, model outputs primarily demonstrate compliance with the prompt structure rather than the simulation of psychological traits. Although these results constrain the applicability of SLMs in psychometric contexts, our proposed framework serves as a valuable diagnostic instrument for detecting detrimental artifacts and isolating true semantic comprehension, thereby aiding future research on frontier models.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC