Title: KnowledgeBerg: Assessing Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models

Abstract:

While many real-world inquiries may seem straightforward at first glance, they often conceal a complex requirement for two distinct abilities: (i) the systematic coverage of a defined knowledge universe and (ii) compositional set-based reasoning within that framework. We refer to this hidden complexity as "the tip of the iceberg." To address this challenge, we define it along two independent axes: knowledge width, which measures the cardinality of the necessary universe, and reasoning depth, indicating the number of compositional set operations involved.

In response, we present KnowledgeBerg, a novel benchmark comprising 4,800 multiple-choice questions. These items were generated from 1,183 enumeration seeds across 10 distinct domains and 17 languages. To guarantee reproducibility, the underlying knowledge universes are anchored in authoritative sources. Our evaluation of representative open-source Large Language Models (LLMs) reveals significant performance deficits, with F1 scores for universe enumeration ranging between 5.26 and 36.88, and accuracy on knowledge-grounded reasoning falling between 16.00 and 44.19.

Diagnostic investigations identify three primary stages where models fail: completeness (omitting required knowledge), awareness (failing to recognize task requirements), and application (executing incorrect reasoning). This failure pattern is consistent regardless of language or model scale. While strategies such as test-time compute and retrieval augmentation provide measurable improvements—yielding gains of up to 4.35 and 3.78 points, respectively—significant performance gaps persist. These results highlight enduring limitations in how contemporary LLMs manage structured knowledge and perform compositional reasoning within bounded domains. The dataset is accessible at https://huggingface.co/datasets/2npc/KnowledgeBerg

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC