Title: The Role of Conversation Topics as Stand-ins for Sociodemographics: The Impact of Context on LLM Responses

Abstract

In high-stakes domains such as law, medicine, and finance, the utilization of large language models (LLMs) reveals a critical vulnerability: even a single thread of conversation history can significantly alter outcomes for different users. While previous studies have linked these variations to outcome disparities among sociodemographic groups—where certain demographics receive more favorable results than others—our research offers a different perspective. We show that LLMs are actually poor at deducing a user’s sociodemographic background from just one conversation. Furthermore, while disparities do exist between these groups, the actual magnitude of these differences is quite small.

To identify the primary cause behind these observed inequalities, we analyzed how various (psycho)linguistic features of dialogue, such as readability, emotional tone, and subject matter, relate to user demographics. Our analysis reveals that the specific topic of conversation is the strongest predictor of the advice generated by LLMs within a given context. Consequently, conversation topics often act as proxies for sociodemographic identity, influencing responses in ways that are difficult to anticipate. This finding raises significant concerns and underscores the urgent need for further investigation into how conversational context shapes LLM outputs, particularly in critical scenarios, and for the development of strategies to mitigate these effects if necessary.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC