Title: Mastering Evaluation: A Low-Cost Approach to Assessing Models on Unlabeled Data via Meta-Learning

Abstract:

As machine learning progresses at a breakneck pace, the proliferation of model ecosystems has made it increasingly challenging to gauge the reliability of newly released systems when faced with unseen, unlabeled data. Traditional evaluation methodologies often depend on expensive human annotation, repetitive fine-tuning processes, or assumptions that fail to generalize across different model types. To address these limitations, we present MetaEvaluator, a model-agnostic framework designed for the rapid, label-free assessment of unseen models spanning various architectures and modalities. By meta-learning from a pool of reference models, MetaEvaluator secures a robust initialization that facilitates accurate evaluations of new models. This approach significantly reduces evaluation costs by amortizing expenses and removes the necessity for per-model retraining. To the best of our knowledge, MetaEvaluator stands as the inaugural model-agnostic framework capable of evaluating new models on unlabeled datasets. Comprehensive experiments confirm that MetaEvaluator provides stable and precise performance estimates at a fraction of the cost associated with conventional methods, thereby facilitating scalable benchmarking for emerging models on unlabeled data. The code is available at: https://github.com/phkhanhtrinh23/MetaEvaluator.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC