Title: BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routine

Abstract:

Breast cancer continues to be a primary driver of cancer-related deaths in women. Managing this disease clinically demands multimodal reasoning throughout a workflow that covers screening, diagnosis, and treatment planning. Each phase of this continuum involves unique imaging types, specific task goals, and distinct reasoning requirements. Yet, due to limitations in data availability and model flexibility, current medical Multimodal Large Language Models (MLLMs) are usually assessed on isolated modalities or narrow task sets, which restricts their capacity to support reasoning at the workflow level.

To address this, we present BreastStage, a new instruction corpus aligned with clinical workflows. This corpus contains 1.86 million instruction-following pairs, curated from 17 sub-datasets spanning five imaging modalities and 136 task templates. We also introduce BreastStage-Bench, a held-out split designed as a comprehensive benchmark to evaluate multimodal reasoning across the entire breast cancer care spectrum.

Leveraging this corpus, we developed BreastGPT, a unified MLLM featuring a dual-branch visual encoder and concept-preserving token compression. These components help bridge the significant scale disparity between standard radiology images and gigapixel pathology slides. In evaluations on BreastStage-Bench, BreastGPT attained a closed-ended accuracy of 75.66% and an open-ended score of 89.92%. These metrics surpass those of both general-purpose and medically specialized MLLMs across various clinical stages and task formats. Our findings indicate that data aligned with clinical workflows and visual modeling capable of handling cross-scale inputs are essential for developing clinically relevant medical MLLMs. All associated data, code, and model checkpoints are publicly available at https://yangyy-liu.github.io/BreastGPT.io.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC