Title: GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answering

Original: arXiv:2605.29584v2 Announce Type: replace

Abstract: Reinforcement learning (RL) is ideally suited for agentic knowledge base question answering (KBQA), a task requiring a model to generate executable actions, interpret feedback from the knowledge base, and ultimately produce an answer. Nevertheless, contemporary RL-driven KBQA systems predominantly focus on optimizing sparse rewards derived solely from the final response, resulting in insufficient supervision for errors occurring during intermediate steps. This limitation is particularly pronounced in logical-form annotated KBQA benchmarks. While gold logical forms can be translated into executable action sequences, existing methodologies primarily utilize them for initializing warm-start data rather than facilitating on-policy RL updates. To address this, we introduce GAPD, a Gold-Action Policy Distillation framework designed to provide dense, token-level guidance during training alongside outcome-based RL. GAPD employs a technique called MID-ANCHOR MATCHING to align gold actions with on-policy student rollouts. This method identifies intermediate entities reached during both student exploration and gold execution as state anchors, matching student states to gold states based on these explored entity sets. The policy conditioned on these aligned gold actions functions as a stop-gradient teacher, with its token distribution being distilled back into the standard student policy across generated action-token spans. GAPD demonstrates consistent superiority over current state-of-the-art methods on the WebQSP, GrailQA, and GraphQ benchmarks.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC