Title: Who Assesses the Societal Consequences of AI? A Landscape Analysis of First- and Third-Party Evaluations

Abstract:

As foundation models become integral to high-stakes artificial intelligence systems, governance structures are increasingly reliant on evaluation metrics to gauge both capabilities and potential risks. While assessments of general technical proficiency are now commonplace, evaluations addressing social implications—such as bias, fairness, privacy concerns, environmental footprints, and labor dynamics—remain inconsistent. This study presents the inaugural comprehensive examination of social impact evaluation reporting, analyzing 186 first-party release documents and 248 third-party evaluation sources, alongside insights gathered from developer interviews.

The analysis reveals a pronounced split in responsibilities. First-party reports are frequently limited in scope, often superficial, and show a downward trend in disclosures regarding environmental impact and bias. In contrast, third-party evaluators offer more extensive and rigorous scrutiny of performance disparities, harmful content, and bias. Despite this, only the developers possess the authority to accurately report on data origins, the labor involved in content moderation, operational costs, and infrastructure details. However, our interviews indicate that developers tend to deprioritize these disclosures unless they are directly linked to regulatory compliance or product adoption. These current practices result in significant blind spots in assessing societal effects, highlighting an urgent need for policies that enforce developer transparency, bolster independent evaluation frameworks, and establish shared infrastructure to consolidate third-party findings.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC