Title: Measuring the Impact of Geo-Cultural Values on Pluralistic Safety Alignment

Abstract

The responsible global rollout of artificial intelligence necessitates that models align with human values, which are not uniform but instead vary significantly across different cultural contexts. However, current safety evaluation datasets predominantly rely on rater pools that are geographically homogeneous, thereby overlooking these crucial geo-cultural distinctions. Furthermore, it is still uncertain whether such cultural differences remain significant when demographic variables like age, gender, and ethnicity are controlled for.

This study conducts a meta-analysis of existing safety datasets, revealing that the majority fail to report geo-cultural data, while those that do lack a standardized framework for jointly analyzing geo-cultural and demographic factors. By applying the Inglehart-Welzel dimensions of cross-cultural variation, we utilize multilevel modeling to show that belonging to a specific cultural zone accounts for additional variance in safety ratings beyond what is explained by standard demographics (p<0.05 across six datasets).

Our analysis also identifies that approximately 10% of the items within the examined datasets are culturally sensitive. These items carry a high risk of being incorrectly classified as safe if the evaluation process lacks sufficient cultural diversity. We further assess the utility of Large Language Models (LLMs) as both proxies for human raters and triage mechanisms. The results indicate that while current LLMs cannot reliably substitute for human raters, they are effective tools for prioritizing culturally sensitive items for human annotation. These insights underscore the need for more culturally pluralistic approaches to safety evaluation and provide actionable recommendations to facilitate this shift.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC