Title: VLESA: A Vision-Language Embodied Safety Agent for Monitoring Human Activities

Abstract:

As artificial intelligence systems become more deeply integrated into physical human tasks, maintaining safety is critical. Unlike digital mistakes, physical errors result in immediate and irreversible consequences. To address this, we present the Vision-Language Embodied Safety Agent (VLESA), a novel framework designed to observe human activities via egocentric video streams and initiate real-time safety interventions when hazardous actions are anticipated. VLESA specifically tackles the challenge of intent-dependent safety, recognizing that the same physical action may be safe or dangerous depending on the surrounding context.

To support this approach, we introduce a new dataset that pairs egocentric video frames with safety annotations conditioned on specific goals. This enables the training of a goal-conditioned safety Q-filter using GRPO, which assesses actions based on inferred intent without requiring model retraining. Additionally, we propose an agent capable of jointly predicting future actions and inferring human goals directly from video input.

Evaluations on the ASIMOV-2.0 benchmark demonstrate that VLESA outperforms baseline methods in intervention accuracy at the precise ground-truth frame. Furthermore, the GRPO-trained Q-filter enhances action safety by more than 41 percentage points, leveraging goal-conditioned constrained decoding. The source code is publicly available at https://github.com/HanjiangHu/VLESA.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC