Title: Identifying Knowledge Deficits in LLM Tool Utilization: An Agentic Framework for Acquiring New APIs

Abstract

Large language models tasked with code generation frequently encounter APIs that were not included in their pretraining corpora. Mastering these tools demands more than simply recalling function names; models must effectively coordinate signatures, module paths, input-output contracts, semantic meanings, and executable usage patterns. Current benchmarks for novel APIs are often limited by their static nature, reliance on simplistic pass/fail metrics, or use of synthetic APIs that fail to mirror the complexity of real-world library evolution.

To address these limitations, we present NovelAPIBench, a fully automated dynamic benchmarking system. This framework operates across any base model and target library to discover novel APIs, extract decomposed knowledge bundles, generate executable coding tasks, and categorize failures into six distinct diagnostic groups. In an evaluation spanning approximately 1,900 tasks, four base models, and five distinct domains, we compared knowledge delivered via retrieval against knowledge internalized through parametric adaptation.

Our findings indicate that different knowledge components are not interchangeable. Usage examples emerged as the most potent standalone signal. The optimal two-component configuration involved pairing signatures with either mechanisms or examples, a choice dependent on the specific domain and backbone model. Furthermore, increasing context volume—particularly by adding source code—can be detrimental, as it tends to increase import-path errors.

Regarding parametric adaptation, we found that it does not eliminate the need for retrieval when external knowledge is removed. Instead, fine-tuning primarily equips models with the ability to utilize provided knowledge bundles, a skill that successfully transfers to unseen libraries. These results highlight a complementary relationship between the two approaches: retrieval serves to supply volatile API content, while tuning enhances procedural integration.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC