PMC-InterCPT: Reimagining Biomedical Interleaved Data for Multimodal Continued Pretraining

arXiv:2606.01049v1

Abstract:

Scientific literature serves as a rich source for large-scale biomedical image-text datasets, which are essential for training medical multimodal models. While these resources are typically structured as isolated image-caption pairs, this format presents significant limitations. Captions are frequently brief, reliant on external context, and insufficiently descriptive when detached from the main article text. Furthermore, automated extraction processes often introduce structural noise, including absent captions, leftover markup, redundant context, and fragmented multi-paragraph descriptions.

To address these challenges, we reevaluate the data construction process for medical multimodal continued pretraining (CPT). We introduce PMC-InterCPT, a context-aware biomedical interleaved corpus that integrates body text referencing figures alongside their corresponding captions. Our methodology involves recovering lost captions, sanitizing both caption and context text, reconstructing coherent interleaved image-text samples, and employing LLM-supervised classifiers to filter out noisy records based on medical relevance and quality.

Our analysis highlights a pronounced modality imbalance within the generated corpus. To mitigate this, we propose a four-bucket evidence taxonomy designed for modality-aware resampling. When applied to continued pretraining and subsequent supervised fine-tuning (SFT) on the Qwen3.5-4B-Base model, PMC-InterCPT demonstrates substantial improvements in both general and medical multimodal performance. Notably, these gains are achieved with fewer CPT tokens than those required by the raw source data. These findings underscore the synergistic relationship between data quality and modality in enhancing medical multimodal CPT.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC