Title: CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks

Abstract:

Selecting or ordering language models for a particular application becomes exceptionally difficult when task-specific labeled data is unavailable and public benchmarks are unreliable. This unreliability stems from the likelihood that benchmark items have contaminated pretraining data, causing performance scores to measure memorization rather than actual capability. To address this challenge, we introduce CoEval, an open-source and reusable framework that provides a complete end-to-end solution. Starting solely with a textual description of a task or domain, CoEval utilizes teacher models to generate a fresh, attribute-controlled benchmark. This process requires no human labeling and ensures the data is contamination-free, as new items are created for every execution. Subsequently, a cross-family judge ensemble ranks candidate models without the need for human evaluators.

When validated against scenarios where ground truth is known, CoEval accurately recovers the true model ranking and maintains a correlation with ground-truth correctness of $h_o=0.86$. The framework’s label-free judging mechanism eliminates the need for human calibration; its reliability is driven by the diversity of the judge panel’s vendors rather than its size. Research indicates that a small, carefully selected cross-family panel yields the highest reliability, whereas relying on a single judge can result in anti-correlation with ground truth (with a judge-choice regret of 0.35). In contrast, the ensemble approach never exhibits this flaw.

Furthermore, the items generated by CoEval demonstrate zero verbatim 13-gram overlap with five major public benchmarks. The panel design effectively neutralizes verbosity bias and prevents same-family self-preference. In a study involving four tasks, the system generated 7,978 evaluations at a cost of just USD 5.89. This declarative pipeline is applicable to any domain and is cost-effective enough to be rerun with every new model release, offering any team the ability to regenerate a label-free, contamination-free leaderboard tailored to their specific needs.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC