Title: Evaluating and Addressing Miscalibration in Social Science Measurement via Large Language Models

Abstract:

Large language models (LLMs) are gaining traction in social science research as scalable instruments for transforming unstructured text into variables suitable for standard empirical frameworks. However, establishing measurement validity requires more than merely achieving high average accuracy; it necessitates well-calibrated confidence that accurately mirrors the empirical likelihood of each measurement’s correctness. This study investigates the phenomenon of model miscalibration within LLM-driven social science measurement.

We start with a case study involving the Federal Open Market Committee (FOMC), demonstrating that when LLM confidence is miscalibrated, filtering based on confidence scores can alter downstream regression estimates. Subsequently, we conduct a comprehensive calibration audit across 14 distinct social science constructs, examining both proprietary models—such as GPT-5-mini and DeepSeek-V3.2—and open-source alternatives. Our findings reveal that, across various tasks and model architectures, the reported confidence levels exhibit poor alignment with correctness as determined by tolerance-based criteria.

To address this issue, we propose a straightforward mitigation strategy: a soft label distillation pipeline designed to calibrate BERT using LLM outputs. This method transforms an LLM’s score and its verbalized confidence into a soft target distribution, which is then used to train a smaller discriminative classifier on encoder-based models. On average across datasets, this technique decreases the Expected Calibration Error (ECE) by 43.2% and the Brier score by 34.0%. These findings indicate that calibration should be regarded as an integral component of measurement validity in LLM-based social science pipelines, rather than being treated as an optional post-processing step.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC