Title: OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agents

Abstract

Developing robust visual web agents demands the ability to reason over long horizons, execute precise grounding, and interact effectively with the fluid nature of live websites. While the field has advanced rapidly, the most powerful systems remain largely closed-source. In contrast, open-source alternatives continue to rely heavily on supervised post-training utilizing extensive collections of curated web trajectories. This reliance imposes a significant scalability constraint: acquiring high-quality demonstrations is costly, and static datasets fail to capture the breadth and constant evolution of the open web. Although online reinforcement learning (RL) has demonstrated value for text-based agents, its application to training visual web agents directly on live sites has received limited attention.

To address this gap, we present OpenWebRL, an open framework designed to train visual web agents using online multi-turn RL on actual websites. The framework encompasses the entire training lifecycle, featuring scalable live-browser infrastructure, supervised initialization, multimodal context handling, trajectory-level success evaluation, and efficient policy optimization for multi-turn interactions. Leveraging this system, we developed OpenWebRL-4B, which sets a new state-of-the-art for open-source models on challenging live-web benchmarks.

Remarkably, with just 0.4K initialization trajectories and 2.2K open-ended RL training tasks, OpenWebRL-4B achieves a 67.0% success rate on Online-Mind2Web and 64.0% on DeepShop. These results surpass previous open agents of comparable or larger size and remain competitive with proprietary offerings such as OpenAI’s CUA and Gemini’s CUA. In addition to benchmark achievements, we conduct a systematic analysis of the critical design choices that render online RL effective for visual web agents, exploring how RL enhances agentic reasoning. Ultimately, this work provides a practical roadmap for creating more capable, reproducible, and cost-efficient open web agents. To facilitate further research, we will publicly release our training data, models, and code.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC