Title: CARE-RL: Leveraging Capability Awareness to Resolve Cross-Domain Conflicts in Reinforcement Learning

Abstract:

While reinforcement learning (RL) equipped with verifiable rewards has driven significant advancements in reasoning-focused Large Language Models (LLMs), scaling this approach to multi-domain environments remains difficult. These challenges primarily stem from the unreliability of rewards in non-verifiable tasks and the interference of capabilities across different domains. To address these issues, we introduce CARE-RL, a framework that integrates protocol-aware reward generation with capability-aware optimization to alleviate cross-domain conflicts.

For tasks lacking verifiable rewards, we employ the Protocol-Aware Generative Reward Model (PA-GRM). This model establishes evaluation protocols and schemas at the prompt level prior to generating trace-conditioned rewards, thereby facilitating task-adaptive yet consistent evaluations of open-ended responses. In the realm of multi-domain optimization, we utilize the Direction-Aware Capability Subspace Projection (DACSP). This technique derives historical capability directions from earlier RL stages and adjusts subsequent updates by amplifying aligned components, suppressing conflicting ones, and retaining orthogonal updates.

Empirical evaluations across benchmarks for mathematics, chat, and instruction-following demonstrate that CARE-RL consistently surpasses standard multi-domain RL baselines. Specifically, the method achieved Total Average scores of 47.9 on Qwen2.5-7B and 50.7 on Qwen3-4B.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC