Title: Pinpointing Robust Social Biases in Large Language Models to Ensure Reliable Conversational Tutoring Systems

Abstract:

The integration of large language models (LLMs) into conversational tutoring agents offers a pathway to scalable, personalized feedback, which has been proven to enhance student outcomes and learning engagement. Nevertheless, these systems carry the risk of reinforcing or exacerbating stereotypical social biases, a concern that is particularly acute within educational environments. This research investigates LLMs in tutoring contexts to isolate "high-confidence social biases"—specifically, scenarios where models fail to recognize biased judgments in instructional dialogues yet remain highly certain in their erroneous evaluations. Such discrepancies can negatively impact the reasoning processes and the quality of feedback delivered to learners.

To address this, we introduce a novel dataset creation methodology designed to facilitate bias assessment under naturalistic instructional conditions. This approach involves regenerating student-AI tutor interactions and inserting turns containing controlled biases sourced from an established benchmark dataset. Leveraging this data, we conduct both computational and human evaluations to gauge multiple LLMs’ capabilities in detecting stereotypical biases, while also analyzing the confidence levels and reasoning mechanisms behind their responses.

Our findings reveal that detecting bias is significantly more difficult in conversational tutoring settings compared to standard benchmark-based tests. Furthermore, state-of-the-art LLMs exhibit a tendency toward overconfidence when making incorrect assessments of stereotypical bias statements. The study demonstrates that model confidence plays a critical role in shaping reasoning and feedback, underscoring the dangers posed by overconfident, biased behavior in LLM-driven tutoring agents. The paper concludes with a discussion on the broader implications, potential mitigation strategies, and avenues for future research.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC