Title: CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions

Abstract:

The assessment of Large Language Models (LLMs) in code generation is fundamentally dependent on the robustness and quality of their test suites. Yet, current benchmarks frequently fail to account for nuanced edge cases, inadvertently permitting flawed solutions to succeed. To address this limitation, we introduce CodeHacker, an automated agent framework designed to produce targeted adversarial test cases that uncover hidden vulnerabilities in program submissions. Emulating the hacking dynamics found in competitive programming, CodeHacker utilizes a multi-pronged strategy comprising stress testing, anti-hash attacks, and logic-specific targeting to dismantle specific code entries.

To guarantee the accuracy and reliability of these attacks, we implemented a Calibration Phase. During this stage, the agent iteratively improves its own Validator and Checker by using self-generated adversarial probes prior to assessing contestant code. Experimental results show that CodeHacker substantially increases the True Negative Rate (TNR) of existing datasets, successfully filtering out incorrect solutions that had previously gone undetected. Additionally, the adversarial cases generated serve as high-quality training data, enhancing the performance of reinforcement learning (RL)-trained models on benchmarks such as LiveCodeBench.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC