Title: HALO: Learning Human-Robot Collaboration via Heterogeneous-Agent Lyapunov Policy Optimization

Abstract:

To enhance the robustness and generalization capabilities of human-robot collaboration (HRC), robots must be equipped to handle a wide array of human behaviors and contextual variations. This necessity drives the adoption of multi-agent reinforcement learning (MARL). However, the fundamental difference between robots and humans introduces a rationality gap (RG), a phenomenon where decentralized policy updates fail to align with cooperative joint optimization. Because this learning scenario constitutes a general-sum differentiable game, standard independent policy-gradient methods often suffer from oscillation or divergence unless additional structural constraints are applied.

In response, we introduce Heterogeneous-Agent Lyapunov Policy Optimization (HALO). This framework stabilizes decentralized MARL by imposing Lyapunov-based contraction within the policy-parameter space. It is important to distinguish HALO from Lyapunov-based safe reinforcement learning, which typically focuses on satisfying state or trajectory constraints within constrained Markov decision processes; instead, HALO leverages Lyapunov certification specifically to stabilize decentralized policy learning. By correcting decentralized gradients through optimal quadratic projections, HALO ensures the monotonic contraction of the rationality gap, thereby facilitating effective exploration across open-ended interaction spaces. Results from comprehensive simulations and real-world experiments involving humanoid robots demonstrate that this certified stability significantly boosts generalization and robustness, particularly in challenging collaborative scenarios. For more details, please visit our project website at https://HaoZhang-THU.github.io/HALO/.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC