Title: VRPRM: Enhancing Process Reward Modeling Through Visual Reasoning

Abstract:

Process Reward Models (PRMs) have become a staple in the post-training phase of Large Language Models (LLMs), primarily due to their ability to conduct granular assessments of the reasoning sequences within generated outputs. Despite their utility, conventional PRMs often struggle with long-term logic and deep analytical thinking. While some recent efforts have attempted to integrate Chain-of-Thought (CoT) capabilities into PRMs, the prohibitive cost of annotating CoT-PRM datasets has hindered their consistent effectiveness across diverse applications.

To overcome these limitations, we introduce VRPRM, a novel process reward model that leverages visual reasoning, supported by an efficient two-stage training framework. Our experiments demonstrate that VRPRM, trained on a modest dataset comprising just 3.6K CoT-PRM Supervised Fine-Tuning (SFT) samples and 50K non-CoT PRM Reinforcement Learning (RL) instances, outperforms non-thinking PRMs that utilize a significantly larger dataset of 400K samples. Notably, VRPRM achieves a relative performance gain of up to 118% over the base model in Best-of-N (BoN) evaluations. These findings validate that our combined training approach fosters superior reasoning quality while minimizing data annotation expenses, thereby establishing a more efficient and scalable paradigm for PRM development.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC