Title: GenPT: Moving Beyond Self-Reports to Achieve Robust LLM Psychometrics Through Generative Projective Testing

Abstract:

While self-report questionnaires continue to dominate the assessment of psychological states in persona-conditioned agents (PC-Agents), traditional instruments are plagued by two significant vulnerabilities: contamination from training data and directional bias stemming from social desirability or contextual framing. To address these methodological constraints, this study investigates the potential of adapting projective paradigms into a reliable psychometric framework. We propose GenPT (Generative Projective Testing), a novel approach that reimagines established tools like the TAT, Rorschach, and SCT by utilizing newly generated stimuli. This method structures the evaluation into a three-stage pipeline designed to extract standardized psychological indicators and target states.

We benchmarked GenPT against classical questionnaires by evaluating PC-Agents configured via CharacterRAG and AnnaAgent profiles. Our findings reveal that traditional questionnaires suffer from systematic directional shifts when subjected to social-desirability framing, with the most pronounced effects observed in assessments of suicide ideation. Conversely, the behavioral patterns captured by GenPT remain closely aligned with symmetric baselines, demonstrating greater stability. Additionally, in a longitudinal counseling scenario, depression assessments conducted via GenPT showed shifts approximately an order of magnitude larger than those from questionnaires when using Qwen3 as the underlying model. These results suggest that GenPT serves as a valuable complement to self-report methods, particularly in contexts where resistance to contamination, mitigation of bias asymmetry, and sensitivity to context are critical. The code and stimuli associated with this research are available at https://github.com/sci-m-wang/GenPT.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC