Title: Transforming Demonstrations into Rewards: Test-Time Prompt Optimization for Vision-Language Model Reward Systems

Abstract: Accurate reward functions are the cornerstone of reinforcement learning, yet in practical domains like robotics, these functions are frequently manually constructed or entirely absent. While recent studies have investigated the potential of pre-trained Vision-Language Models (VLMs) as zero-shot reward models, this approach is vulnerable to suboptimal outcomes if prompt engineering is not meticulously managed. Specifically, high rates of false positive predictions can significantly hinder downstream policy learning. In robotic contexts, policy training is often initiated using small datasets of expert demonstrations. This setup offers a unique chance to refine the reward model before the main training phase begins. To address this, we introduce Demo2Reward, a test-time adaptation method that optimizes the language instructions of a reward model using a minimal set of demonstrations (3–10 trajectories). This technique aims to minimize false positives without sacrificing true positives. Notably, it achieves these improvements without requiring additional model training or computational overhead during policy learning. Our experiments demonstrate that Demo2Reward consistently surpasses current zero- and few-shot VLM reward models across various simulated robotic tasks and different policy backbones. Furthermore, we validate the method’s effectiveness in a real-world robotic learning environment, showing that it facilitates policy learning without the need for manual reward function design.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC