Title: Leveraging Large Language Model Encoders to Enhance Image-Text Retrieval in Chest Radiography

Abstract:

Effective cross-modal alignment is a fundamental requirement for scalable analysis and retrieval in medical data-driven informatics, making multimodal learning from paired clinical text and medical images a primary challenge. In the specific context of chest radiography, vision-language pretraining faces significant hurdles due to the heterogeneous nature of radiology reports, which often feature abbreviations, impression-only summaries, and writing styles unique to specific institutions. In contrast to general-domain applications, simply aggregating large volumes of noisy reports can lead to performance plateaus or even deterioration in multimodal learning, particularly when reporting styles vary significantly.

To address this, we introduce a domain-adapted bidirectional large language model (LLM) text encoder tailored for chest radiograph reports. This encoder is trained using masked token prediction alongside supervised contrastive learning, leveraging stylistically diverse yet clinically equivalent report variants to generate robust and generalizable text embeddings. Subsequently, we incorporate this encoder into a dual-tower contrastive vision-language framework, employing parameter-efficient adaptation to enhance image-text alignment.

Evaluated across 1.6 million paired studies drawn from public datasets and a de-identified hospital cohort, the proposed models demonstrate improved bidirectional retrieval accuracy and superior external generalization. The system achieved GREEN scores of 0.308 on MIMIC-CXR and 0.618 on Open-I. Notably, the approach mitigates the performance degradation typically observed when incorporating abbreviation-heavy, impression-only hospital reports into the training process.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC