Title: Evaluating Generalizability, Agreement, and Validity in French Automated Essay Scoring and Language Certification

Abstract:

While Automated Essay Scoring (AES) benchmarking has often encouraged reductive evaluation methods, broader assessment frameworks like the argument-based validation (ABV) approach advocate for a more comprehensive, multidimensional analysis of systems, particularly within high-stakes language testing environments. This study presents a refined and more applicable iteration of the ABV framework. This updated version integrates fairness assessments, correlations with linguistic characteristics, prediction error metrics, and comparisons of model alignment with human raters. We apply this enhanced framework to AES for the French language, evaluating eight distinct model architectures. The testing utilizes a dataset comprising 27,000 exam essays, each scored by two raters, alongside a generalization corpus of 961 essays, each evaluated by at least nine raters. Our findings highlight how employing the ABV framework provides deeper insights into the strengths and limitations of AES models, thereby contributing to advancements in French AES capabilities.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC