Title: IndoBias: A Dual-Track, Culturally Anchored Benchmark for Assessing LLM Bias in Indonesian Contexts

Abstract:

Indonesia’s sociocultural landscape is characterized by its remarkable diversity, encompassing over 1,300 ethnic groups and 700 indigenous languages. Despite this richness, the issue of bias within Large Language Models (LLMs) remains largely unexplored in the region, creating a significant void in our ability to evaluate representational fairness and localized stereotypes. To bridge this gap, we present IndoBias, a benchmark rooted in cultural context designed to measure LLM bias in both Indonesian and three regional languages: Javanese, Sundanese, and Makasar.

IndoBias employs a dual-track evaluation methodology. The first track is depth-oriented, utilizing contrastive pairs, while the second is breadth-oriented, relying on generation-based tasks grounded in established social science frameworks, specifically SPI, O*NET, and WGI.

Our analysis reveals distinct bias patterns across different models and languages. Existing LLMs, particularly decoder-based architectures, demonstrate a pronounced bias toward prototypical sentences in Indonesian. Conversely, models show higher levels of bias in the categories of Ideology and Religion when processing local languages. Furthermore, we observed that LLM responses display non-uniform Stereotype Polarity when exposed to various local entities.

In terms of training data, we found that pretraining on Common Crawl texts introduces more bias into Indonesian models compared to using human-reviewed sources such as Wikipedia or news articles. Additionally, incorporating local languages into the pretraining phase generally leads to an increase in bias. This study underscores the necessity of examining bias within specific cultural frameworks.

Warning: This paper includes example data that may be considered offensive, harmful, or biased.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC