Title: Can Vision Language Models Acquire Intuitive Physics Through Interaction?

Abstract: Pre-trained vision-language models currently lack robust intuitions regarding the physical world. Although recent studies indicate that supervised fine-tuning enhances performance on basic physical tasks, these adjusted models do not seem to acquire stable physical rules capable of generalizing to novel scenarios. Drawing from cognitive science, we propose that environmental interaction is essential for models to accurately grasp physical dynamics. We evaluated this hypothesis by training models via reinforcement learning within a simulated environment. Our results demonstrate that while interaction boosts performance within specific tasks, it does not yield generalizable physical intuition. Specifically, we observed that models trained on one task fail to reliably transfer their knowledge to related tasks, regardless of shared visual statistics or physical principles, and irrespective of whether the training method involved direct interaction.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC