Title: Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams

Abstract

Vision-language models encounter a unique set of obstacles when processing engineering diagrams. Unlike standard natural images or conventional documents, these diagrams rely on dense spatial arrangements, specialized domain symbols, and intricate cross-references between visual callouts and structured parts tables. Although such diagrams are vital to service, repair, and design processes, the field currently lacks a public benchmark to assess vision-language model (VLM) proficiency in this area; existing resources tend to concentrate on flowcharts, scientific illustrations, or corporate paperwork.

To bridge this void, we present Enginuity, the inaugural open dataset and benchmark designed to evaluate VLMs on intricate engineering diagrams. Our benchmark comprises two distinct tasks derived from a corpus of U.S. military service and repair manuals: Task 1 involves extracting structured parts tables, while Task 2 focuses on free-form visual diagram question answering (VQA). We assessed four leading VLMs—GPT-5.2 Chat, Claude Opus 4.7, Gemma 4, and Qwen3-VL-32B-Instruct—using both zero-shot and chain-of-thought prompting strategies.

The results for Task 1 indicate that while models achieved Recall@all scores ranging from 0.61 to 0.87, their Token F1pen scores were significantly lower, falling between 0.03 and 0.18. This disparity highlights a systematic disconnect between the ability to identify parts and the accuracy of their descriptions. Meanwhile, Task 2 uncovered a persistent factual-reasoning gap among all evaluated models.

Our supplementary analysis demonstrates that token-overlap metrics underestimate model performance on technical descriptions by a factor of 2 to 6 compared to semantic similarity measures. This finding underscores the necessity of employing LLM-as-judge calibration for more accurate, domain-specific evaluations. To facilitate reproducible research into VLM capabilities regarding engineering content, we have made the dataset, annotations, evaluation harness, and per-sample model outputs publicly available.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC