Title: Using LLMs as Meta-Judges to Validate NLP Evaluation Metrics with Synthetic Data

Abstract: The conventional validation of natural language generation (NLG) evaluation metrics is often hindered by the high costs and time demands of human annotation, a resource that is largely restricted to English-language datasets. To address this, we introduce "LLM as a Meta-Judge," a scalable framework that employs large language models to create synthetic evaluation datasets. This process involves applying controlled semantic degradation to real-world data, effectively substituting the need for human judgment. We assess the efficacy of this method through "meta-correlation," which quantifies the consistency between metric rankings generated from synthetic data and those established by standard human benchmarks. Our experiments, spanning Machine Translation, Question Answering, and Summarization tasks, indicate that synthetic validation is a trustworthy substitute for human assessment. Notably, the approach achieved meta-correlations above 0.9 in multilingual QA scenarios, confirming its viability as a practical alternative in contexts where human evaluation is either inaccessible or prohibitively costly. The associated code and datasets will be made publicly available once the paper is accepted.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC