Title: SWE-rebench V2: A Scalable, Language-Agnostic Collection of SWE Tasks

Abstract:

The rapid advancement of software engineering (SWE) agents has been significantly propelled by recent progress in reinforcement learning (RL). However, the effectiveness of RL training is currently hindered by a lack of extensive task collections that feature reproducible execution environments and dependable test suites. While the number of benchmarks is increasing, available datasets for training purposes often suffer from limited scale and diversity, frequently focusing only on high-resource language ecosystems.

To address these limitations, we present SWE-rebench V2, a language-agnostic automated pipeline designed to harvest real-world, executable SWE tasks and construct RL training environments at scale. This pipeline utilizes an interactive setup agent to synthesize repository-specific installation and testing procedures. It further employs an ensemble of LLM judges to filter out unsound instances, a process validated against human-verified SWE-bench annotations.

Leveraging this pipeline, we have compiled a dataset comprising 32,079 tasks across 3,617 repositories and 20 different programming languages, complete with pre-built images to ensure reproducible execution. To expand the volume of training data, we also release over 120,000 tasks. These entries include installation instructions, fail-to-pass tests, and rich metadata, with problem statements generated from the original pull request descriptions.

We validated the collected instances through a diagnostic study involving a subset of tasks in five programming languages and seven popular models. Additionally, we provide instance-level metadata to highlight common confounders, such as overly restrictive tests and underspecified descriptions. By releasing the datasets, collection and execution code, and associated artifacts, we aim to facilitate large-scale training of SWE agents across a diverse range of languages and repositories.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC