Title: TurtleAI: Evaluating Multimodal Models on Visual Programming in Turtle Graphics

Abstract

While Vision-Language Models (VLMs) have increasingly been applied to visual programming to generate code for visual tasks, existing research has predominantly targeted productivity applications. Consequently, it remains uncertain how effectively current VLMs handle education-focused visual programming or what specific constraints hinder their performance. To address this knowledge gap, we present TurtleAI, a new benchmark comprising 823 tasks derived from authentic visual programming scenarios within the Turtle Graphics environment. Successfully completing these tasks demands that models interpret geometric patterns, deduce spatial relationships, and produce Python code that accurately recreates those patterns.

We assessed the capabilities of over 20 VLMs, including GPT-5, GPT-4o, and Qwen2-VL-72B, discovering that they face substantial challenges, with the majority achieving success rates under 30%. To mitigate these shortcomings, we developed a data generation method requiring only a limited number of seed samples. When Qwen2-VL-72B was fine-tuned on the resulting synthetic dataset, it demonstrated an approximate 20% performance boost on real-world tasks. Our analysis of failures indicates that GPT-4o is particularly weak in spatial reasoning and exact visual replication. Furthermore, we found that fine-tuning primarily enhances the correlation between a model’s visual reasoning processes and its code implementation.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC