Title: Creating an Automated VQA Benchmark for Oncology from Reports to Evaluate Vision-Language Models on 3D Medical Imaging

Abstract:

To properly assess vision-language models (VLMs) in the medical domain, evaluation frameworks must be scalable, free from evaluation confounds, and rooted in clinical reality. Current public benchmarks suffer from constraints such as limited scale, reliance on manual annotation, or the risk of data leakage into VLM pretraining datasets. To address these gaps, we introduce an automated, agent-driven pipeline that constructs multiple-choice Visual Question Answering (VQA) datasets by leveraging paired private radiology reports and 3D oncology imaging. This process generates two distinct categories of questions: RADS-style queries, which are deterministically extracted based on clinician-defined reporting schemas, and questions derived from radiology reports, which are formulated by a Large Language Model (LLM) from radiologist findings and validated against the source documentation.

When applied to four internal cancer cohorts, this methodology produces a benchmark that controls for instance contamination without requiring per-question human oversight. Our zero-shot evaluation of six VLMs demonstrates that no single model currently dominates, with significant performance gaps remaining across all categories. Furthermore, a blind ablation study highlights that dependence on visual input is highly specific to the dataset: while questions derived from liver reports genuinely necessitate image analysis, Lung CT questions are often solvable without visual data. Notably, the top-performing closed-source model actually achieved higher accuracy when blinded to the images on Lung CT tasks. This finding suggests that even private clinical data cannot ensure a contamination-controlled assessment of visual capabilities. The pipeline is now available as an open agent skill to facilitate in-house redeployment.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC