Title: Balancing Privacy and Pedagogy: Addressing Numeric Ambiguity in the MathEd-PII Benchmark for Math Tutoring De-Identification

Abstract:

While the extensive exchange of dialogue data is essential for progressing the field of educational science, strict de-identification protocols often hinder this progress. In transcripts of mathematics tutoring sessions, numerical expressions often mimic the format of structured identifiers, such as dates or personal identification numbers. This similarity causes standard Personally Identifiable Information (PII) detection tools to incorrectly flag and remove essential instructional material, thereby diminishing the value of the data. This study explores methods for identifying PII without compromising educational utility, specifically targeting the challenge of "numeric ambiguity."

To address this, we present MathEd-PII, a novel benchmark dataset designed for PII detection within math tutoring conversations. This dataset was developed using a human-in-the-loop approach with Large Language Model (LLM) annotation. Our analysis, which employs density-based segmentation, reveals that erroneous PII redactions are concentrated in sections with high mathematical density. This finding confirms that numeric ambiguity is a primary source of failure for current detection systems.

Furthermore, we evaluate four distinct detection strategies: a standard Presidio baseline and three LLM-based methods utilizing basic, math-aware, and segment-aware prompting techniques. The results indicate that domain-specific prompting significantly enhances performance. Both the math-aware approach (achieving an F1 score of 0.802) and the segment-aware variant (F1: 0.821) markedly outperform the baseline model (F1: 0.379). These improvements demonstrate a reduction in false positives related to numbers, proving that incorporating domain context is crucial for maintaining the analytic utility of tutoring data during de-identification. This research offers a new benchmark and provides evidence that preserving utility in tutoring data de-identification necessitates domain-aware modeling.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC