Title: PoliticsBench: Benchmarking Political Values in Large Language Models with Multi-Turn Roleplay

Abstract

As Large Language Models (LLMs) become primary information sources, concerns regarding their potential political bias and its effect on objectivity have grown. Current benchmarks for social bias in LLMs largely focus on demographic stereotypes, and when political bias is assessed, it is typically done at a broad level, missing the underlying values that drive sociopolitical reasoning. To address this gap, we present PoliticsBench, a multi-stage roleplay benchmark designed to evaluate fine-grained value expression in LLMs.

In our study, models navigated twenty dynamic scenarios, articulating tradeoffs, adopting positions, and making decisions amidst competing pressures. Testing eight prominent LLMs revealed that scenario-based prompting generates broader and more pronounced value profiles compared to direct political questioning. Specifically, peak interaction stages saw an increase of approximately 0.75 in the number of strongly activated value dimensions (out of a total of 10), a statistically significant rise compared to baseline prompting ($p < 0.05$).

Furthermore, we observed that commitment to a chosen stance intensified throughout the interaction, climbing by roughly 1.4 points on a $[0,5]$ scale from the initial to the decision stages. Although responses in later interaction phases became less robust to paraphrasing of the scenarios, inter-judge agreement remained relatively consistent. These findings indicate that assessing LLM political behavior necessitates a shift from static prompts to extended interactive environments that reflect how values are applied within specific contexts.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC