Title: Gate AI: Methodology and Outcomes of LLM Security Benchmark Evaluation

Abstract:

Current assessments of prompt-injection and jailbreak detection systems for Large Language Models frequently exhibit two critical flaws: the tuning of thresholds on a per-dataset basis and the lack of transparency regarding specific operating points. This paper introduces an evaluation framework designed to rectify these issues. We assess detector performance across 16 public benchmarks, comprising 12,111 samples, utilizing a 5-fold cross-validation approach. The primary validation method employs StratifiedKFold at the row level. Additionally, a parallel diagnostic using StratifiedGroupKFold is conducted over a composite key—defined by parent-prompt ID combined with MinHash and LSH near-duplicate clusters at a Jaccard similarity of $\gtrsim 0.8$—to identify potential data leakage.

To ensure consistency, a single global operating point is determined using held-out folds, specifically maximizing F1 score while maintaining a false-positive rate (FPR) of no more than 1%. This uniform threshold is then applied across all datasets, ensuring that reported results stem from a single decision boundary rather than benchmark-specific optimization. We rigorously test generalization capabilities through a comprehensive suite of diagnostics, including leave-one-dataset-out cross-validation, random-label controls, adversarial validation, permutation feature importance, length-bias correlation, classifier-head agreement, cross-source near-duplicate detection, threshold transferability, train-vs-out-of-fold agreement, and a paraphrase-invariance probe. Each diagnostic includes a quantitative pass threshold, while those that fail have their specific failure modes documented.

For all external comparisons, we re-tune the detector’s threshold to align with the competitor’s published false-positive rate. This ensures that head-to-head evaluations are conducted at matched operating points, providing a fair and standardized comparison.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC