Title: PaintBench: A Deterministic Framework for Assessing Precise Visual Editing

Abstract:

Although contemporary multimodal models demonstrate considerable skill in open-ended visual editing, achieving precise, single-answer edits continues to present a significant hurdle. To investigate this difficulty, we present PaintBench, a dynamically scalable benchmark designed to assess 20 core precise visual editing operations. These operations are organized into four distinct categories: geometric transformation, structural manipulation, color modification, and symbolic reasoning. By utilizing procedural generation with adjustable complexity, PaintBench creates an effectively limitless evaluation suite that is resistant to data contamination. Furthermore, it employs deterministic pixel-level assessment, thereby removing the dependence on judge models that are susceptible to bias.

Our evaluation of 11 image editing models reveals generally low performance levels; notably, the leading industry model achieved a maximum score of only 17.1% (mIoU). Analysis through task decomposition highlights specific operational difficulties, particularly in geometric transformation, most forms of structural manipulation, and formula-based color changes, while also identifying model-specific specializations. Detailed diagnostics from the benchmark indicate that performance declines are driven by variations in scene characteristics, including object quantity, background intricacy, color palette, and the size of the region being edited.

To validate whether PaintBench scores generalize to real-world application performance, we developed a procedural, deterministic evaluation for data visualization editing known as TinyGrafixBench. We observed a strong linear correlation between the results on TinyGrafixBench and PaintBench ($R^2 = 0.91$, $p < 0.001$). Collectively, PaintBench establishes a rigorous standard for measuring advancements and driving progress in precise multimodal visual editing.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC