Title: Identifying Competency Deficiencies in Large Language Models and Their Evaluation Frameworks

The assessment of large language models (LLMs) is predominantly dependent on standardized benchmarks. While these tools offer valuable aggregated performance metrics, they often mask two critical issues: (i) specific sub-domains where models exhibit weaknesses, referred to as "model gaps," and (ii) uneven representation within the benchmarks themselves, known as "benchmark gaps."

To detect these discrepancies automatically, we introduce a straightforward methodology leveraging concept activations derived from sparse autoencoders. This approach enables the identification of fine-grained gaps on a per-concept basis. By grounding the evaluation process in the model’s internal representations, the method facilitates straightforward comparisons across different benchmarks.

We demonstrated the utility of this technique by applying it to five widely used open-source models and over a dozen distinct benchmarks. Validation of the approach confirmed that our unsupervised, automatic system could successfully reproduce model gaps previously identified in academic literature, such as those concerning sycophancy, while also revealing new, previously undocumented deficiencies. Furthermore, the method automatically detected benchmark gaps by pinpointing core concepts that ought to be included within a benchmark’s scope.

The "competency gaps" framework serves as a complementary tool to existing benchmarks. It achieves this by offering a concept-level breakdown of model behavior and assisting developers in refining benchmark design. The source code for this method is accessible at https://competency-gaps.github.io.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC