Title: StateVLM: A State-Aware Vision-Language Model for Robotic Affordance Reasoning

Original: arXiv:2605.03927v2 Announce Type: replace

Abstract: While Vision-Language Models (VLMs) have demonstrated significant efficacy in executing diverse robotic tasks by interpreting visual data and comprehending natural language commands, their deployment in robotics is hindered by a core deficiency shared with Large Language Models (LLMs): poor numerical reasoning. This weakness is particularly evident in tasks requiring object detection and the localization of object states. To address this, we introduce a specialized training methodology that reframes numerical reasoning as a regression problem within VLMs, thereby enhancing their capabilities in object detection and state localization. Our method integrates an Auxiliary Regression Loss (ARL), calculated from box decoder outputs, during the fine-tuning phase, while maintaining standard sequence prediction protocols during inference. Utilizing this strategy, we developed StateVLM (State-aware Vision-Language Model), a framework engineered to capture detailed object representations, including precise object and state localization, as well as identifiable graspable areas. To facilitate research in this area, we present OSAR (Object State Affordance Reasoning), an open-source benchmark comprising 1,172 scenes, 7,746 distinct objects, and their respective bounding boxes, created to fill the gap in existing object-state affordance reasoning datasets. Comparative evaluations on established benchmarks (RefCOCO, RefCOCO+, and RefCOCOg) reveal that incorporating ARL boosts model performance by an average of 1.6% relative to models lacking this loss function. Further testing on the OSAR benchmark confirms these results, indicating that StateVLM equipped with ARL outperforms non-ARL counterparts by an average margin of 5.2%. Notably, ARL plays a critical role in the intricate task of affordance reasoning within OSAR, significantly improving the consistency of the model’s outputs.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC