Title: Vision-Language Models Lack the Capacity to Reason Through Physical Transformations

Abstract: The ability to comprehend physical transformations is a cornerstone of reasoning within dynamic settings. Although Vision Language Models (VLMs) have demonstrated potential in embodied contexts, it remains uncertain whether they truly grasp the mechanics of such transformations. To address this gap, we present ConservationBench, a benchmark designed to assess conservation—the principle that physical quantities remain constant despite transformations. Our evaluation covers four distinct properties, utilizing paired scenarios that either conserve or do not conserve these quantities, resulting in the generation and assessment of 23,040 questions across 112 different VLMs.

The results highlight a pattern of systematic failure: model performance hovers around chance levels. Notably, any gains observed in conservation tasks are offset by declines in control tasks. Further control experiments indicate that models rely heavily on textual priors that assume invariance; however, when visual input is introduced and performance is balanced between conserving and non-conserving cases, their accuracy actually decreases. Strategies such as temporal resolution, specific prompting techniques, or curated sampling methods fail to mitigate these issues. Ultimately, these findings suggest that current VLMs are unable to preserve transformation-invariant representations of physical attributes throughout dynamic scenes.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC