Title: VulnAgent-R2: Evidence-Calibrated Multi-Agent Auditing for Repository-Level Vulnerability Detection

Abstract:

Isolated function classifiers often yield fragile and poorly calibrated warnings because software vulnerabilities frequently hinge on cross-file data flows, build configurations, framework conventions, and runtime guards. While repository-level LLM agents have the potential to gather richer evidence, previous iterations have failed to adequately specify reproducibility, verifier behavior, baseline fairness, and statistical uncertainty. To address these gaps, we introduce VulnAgent-R2, a budget-aware agentic auditing framework that incorporates three reusable modules: counterfactual evidence reweighting, build-aware verification-plan synthesis, and a cost-risk Pareto scheduler. The system integrates graph triage, bounded context optimization, role-specialized agents, skeptical counter-evidence, selective dynamic verification, and calibrated fusion.

Performance evaluations on the Devign, Big-Vul, DiverseVul, and PrimeVul datasets show F1/AUROC scores of 0.798/0.895, 0.739/0.871, 0.700/0.842, and 0.385/0.781, respectively. On the JITVul dataset, the model achieved a 0.606 F1 score, 0.529 Top-1 localization, and 0.742 Top-3 localization, while cutting online token usage by 38.3% compared to always-full multi-agent execution. Note that online time encompasses retrieval, LLM calls, CER scoring, verifier planning, compilation, and test execution, but excludes one-time shared indexing. Bootstrap analysis indicates that the performance gain of VulnAgent-R2 over VulnAgent-X on PrimeVul is +0.038 F1 (95% CI [0.020, 0.055], Holm-adjusted $p=0.009$). By treating vulnerability detection as a process of calibrated evidence accumulation, our approach enhances detection, localization, auditability, and cost control. However, it remains a prioritization aid rather than a substitute for manual review. The code is available at https://github.com/renweimeng/Vlun-Agent-X.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC