Title: Bridging the Gap Between Observation and Execution: Advancing Multimodal GUI Reasoning via a UI-in-the-Loop Framework

Abstract: Graphical User Interface (GUI) reasoning continues to present significant hurdles, with UI comprehension remaining a particularly difficult obstacle. Standard methodologies often depend on immediate decisions derived from screen captures; however, this approach suffers from a lack of transparency and fails to capture a holistic view of interface components, frequently resulting in unsuccessful task outcomes. To address these limitations and improve both the comprehension of and interaction with UIs, we introduce a novel reasoning framework known as UI-in-the-Loop (UILoop). This paradigm redefines GUI reasoning as an iterative cycle involving screens, UI elements, and actions. By empowering Multimodal Large Language Models (MLLMs) to explicitly acquire knowledge regarding the localization, semantic purpose, and functional application of critical UI components, UILoop facilitates accurate element identification and supports transparent, interpretable reasoning. Additionally, we define a more rigorous UI Comprehension task focused on UI elements, supported by three distinct evaluation metrics. In conjunction with this, we release UI Comprehension-Bench, a benchmark comprising 26,000 samples designed to thoroughly assess current methods' proficiency in handling UI elements. Our extensive experimental results indicate that UILoop not only sets new state-of-the-art standards for UI understanding but also delivers enhanced performance across various GUI reasoning tasks.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC