Title: Framing Over Facts: A Communicative Audit of How LLMs Respond

Abstract:

As large language models (LLMs) become more prevalent in addressing subjective, information-seeking queries, users are increasingly prioritizing the manner of communication over mere factual accuracy. However, current evaluation frameworks for LLMs tackling subjective cultural questions tend to concentrate exclusively on factual precision, overlooking the critical aspect of response framing. To address this gap, we present FRANZ, an automated FRAmework for respoNse characteriZation designed to perform a comprehensive communicative audit of LLM outputs across four specific dimensions: adherence to conversational maxims, use of generalizing language, anthropomorphic cues, and cultural positioning.

To support this evaluation, we introduce SQUARE, a new corpus comprising 376,000 subjective questions drawn from 57 different subreddits. These queries are categorized into 19 distinct types and mapped to seven countries. We validate the utility of FRANZ by applying it to generate scores for responses from three open-weight LLMs. Our analysis reveals that these models exhibit statistically significant variations in their propensity to utilize specific response characteristics. Furthermore, unlike audits that rely on single dimensions, FRANZ uncovers a positive correlation between insider positioning and anthropomorphism—a relationship whose intensity fluctuates depending on the country. This finding offers a diagnostic tool for pinpointing divergences in how LLMs frame their responses.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC