Title: Rethinking AI Benchmarks for Resource-Constrained Settings: Moving Past Leaderboard Metrics

Abstract:

Current approaches to evaluating artificial intelligence frequently overlook how these systems function in low-resource environments, where practical operational limitations are just as decisive as raw model quality. By conducting a systematic review of benchmark families covering speech processing, chat/RAG systems, and vision technologies, this study highlights significant discrepancies between controlled laboratory assessments and the realities of deployment in resource-constrained areas. We contend that the appropriate unit for evaluation is the fully deployed system, not a standalone model. Consequently, robust evaluation frameworks must combine task performance metrics with specific deployment variables, including noisy audio inputs, code-switching, unstable connectivity, low-end hardware constraints, and domain shifts. Furthermore, benchmarks must acknowledge that various application categories demand distinct evaluation profiles, rather than relying on a single aggregate score that masks critical operational nuances. To facilitate informed decision-making, we introduce a standardized reporting framework that maintains cross-system comparability and applicability across different contexts while remaining attuned to deployment specifics. Finally, we stress the importance of creating clear, actionable reporting tools for policymakers, funders, and practitioners. These tools should include standardized one-page benchmark summaries, detailed deployment profiles, and explicit documentation of failure protocols and human oversight procedures.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC