Title: SEA-NLI: Using Natural Language Inference to Probe Southeast Asian Cultural Comprehension

Abstract: While large language models (LLMs) demonstrate strong capabilities in Western settings, they are rarely evaluated on underrepresented regions, including Southeast Asia (SEA). Current Natural Language Inference (NLI) benchmarks are predominantly centered on Western cultures, often relying on translations or single-language datasets, which restricts their capacity to assess reasoning that is deeply rooted in specific cultural contexts. To address this gap, we present SEA-NLI, a culturally embedded NLI benchmark that encompasses eight Southeast Asian nations. The dataset is available in both English and native regional languages and has been validated by native speakers. Our evaluation across 17 encoder and decoder models reveals consistently low performance across the board, with particularly poor results in knowledge-heavy domains like Languages and Science and Technology. Further analysis indicates that these shortcomings are primarily due to a lack of SEA-specific cultural knowledge. While adapting models to the region and employing culture-aware prompting strategies yielded improvements, Chain-of-Thought (CoT) prompting provided only marginal benefits.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC