Title: SPADER: Enhancing Multi-Answer Question Answering via Step-wise Peer Advantage and Diversity-Aware Exploration Rewards

Abstract

Large language models are increasingly being utilized as agents augmented with tools to access information that extends beyond their internal parametric knowledge. Although recent advancements have enhanced reasoning capabilities for long-horizon tool usage, the majority of existing methods are designed for tasks requiring a single correct response. However, numerous real-world inquiries demand the identification of a comprehensive collection of valid answers, a scenario defined as Multi-Answer Question Answering (QA). This specific setting introduces two primary difficulties: assigning credit with fine granularity across extended search paths and aligning rewards to encourage sustained exploration, particularly beyond commonly encountered, high-frequency entities.

To address these issues, we introduce SPADER, a reinforcement learning framework tailored for long-horizon tool use within the context of Multi-Answer QA. SPADER incorporates Step-wise Peer Advantage (SPA), a mechanism for step-level credit assignment that operates without a critic. This approach aligns parallel trajectories based on decision steps and calculates advantages by leveraging peer returns. Additionally, the framework features a diversity-aware exploration reward designed to facilitate the discovery of long-tail entities by increasing the weight of rare findings while decreasing the weight of redundant ones.

Evaluations conducted on the QAMPARI, Mintaka, WebQSP, and QUEST benchmarks demonstrate that SPADER generally achieves higher recall and overall F1 scores compared to prompting-based agents, outcome-supervised reinforcement learning methods, and recent techniques employing step-level supervision. The source code and model weights are accessible at https://github.com/KhanCold/spader.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC