Title: Done, Yet Uncertain: Separating World Completion from Self-Termination in Embodied Agents

Abstract:

Conventional embodied evaluation metrics fail to distinguish whether an agent genuinely commits to finishing a task at the end of an episode—a capability we define as terminal commitment. Consequently, behaviorally distinct errors, such as failing to complete the task entirely, executing the task but failing to halt, or claiming success without adequate proof, are all lumped together as simple benchmark failures. To address this, we present VIGIL, an evaluation framework designed to measure terminal commitment as an independent metric.

Under VIGIL’s standard protocol, agents operate using only egocentric RGB visual input and receive no feedback regarding action success. They are required to conclude each episode with a semantic report, which is then deterministically verified against a hidden world state. This methodology generates two distinct scores: world-state completion (W) and benchmark success (B). While W measures task achievement, B is only awarded if the agent also provides a correct terminal report.

This separation allows for the identification of four specific outcome categories: missed execution, drift after goal attainment, unsupported commitment, and verified success. In tests involving 20 models across 1,000 frozen episodes, we observed that systems with similar W scores could differ by as much as 19.7 percentage points in their B scores. For instance, one model successfully translated achieved states into accurate reports, whereas another with nearly identical execution capabilities failed to close the episode, drifting past the goal.

Furthermore, an intervention involving action feedback confirmed this distinction: while execution-oriented signals broadly improved W, commitment failures remained prevalent in models that had not already grounded their terminal reports in the realized state. VIGIL offers a protocol that renders terminal commitment both visible and independently scorable.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC