Title: Enhancing FOLIO and MALLS: A Framework for LLM-Assisted Human Relabeling and Verified Annotations

The integrity of Natural Language to First-Order Logic (NL-to-FOL) benchmarks is critical for the advancement of Natural Language Inference (NLI) and neurosymbolic AI, as these systems rely on precise translations. Despite this importance, previous studies have lacked rigorous auditing of such datasets. This paper addresses that gap through two primary contributions.

First, we conduct a comprehensive human inspection of the validation split of \textsf{FOLIO} and a selected subset of \textsf{MALLS} test instances. Our analysis reveals significant quality issues: approximately 39% of \textsf{FOLIO} entries and 36% of \textsf{MALLS} entries contain flawed First-Order Logic formalizations (ground truth labels). Furthermore, we identified ambiguous natural language sentences in 16.4% of \textsf{FOLIO} cases and 48% of \textsf{MALLS} cases, alongside incorrect NLI labels in 8.4% of \textsf{FOLIO} instances.

Second, we generate and publish corrected ground truths for these datasets. Our results demonstrate that existing annotation errors significantly skew model evaluations on reference benchmark tasks. When we re-evaluated three leading large language models—Gemma~4 31B-it, Qwen3-30B-A3B, and GPT-4o-mini—using the corrected labels, we observed accuracy improvements ranging from 9 to 22 percentage points.

Building on these insights, we introduce an LLM-driven framework designed to assist humans in manually reviewing NL-to-FOL datasets. By guiding reviewers to focus on the most error-prone instances, our approach proves highly efficient. Empirical evidence shows that this targeted method achieves 90% dataset accuracy after reviewing less than 24% of the data, whereas unguided review requires examining over 70% of instances to reach a similar level of quality. We have made all human-verified annotations and the code for our framework publicly available.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC