Title: ES-Merging: Merging Biological MLLMs Using Embedding Space Signals

Abstract:

Biological multimodal large language models (MLLMs) have recently established themselves as potent foundational tools for scientific advancement. Despite their potential, current models are typically confined to individual modalities, which hinders their capacity to address scientific challenges that inherently require cross-modal integration. Although model merging offers a computationally efficient strategy for synthesizing distinct modalities into a cohesive MLLM, conventional approaches depend on input-agnostic heuristics within the parameter space. These methods often fail to accurately reflect the specific specializations of each modality.

To address this shortcoming, we introduce Embedding-Signal-based MLLM Merging (ES-Merging). This framework shifts the merging paradigm from parameter signals to embedding signals by deriving merging coefficients from the latter. ES-Merging leverages both coarse-grained and fine-grained data from the embedding space to calculate layer-wise and element-wise merging coefficients, respectively. These two types of coefficients are integrated to provide a complementary estimation process.

Our extensive experimental results show that ES-Merging surpasses existing merging techniques in both single-modal knowledge retention and cross-modal reasoning. These findings confirm that utilizing embedding space signals offers a robust and principled basis for the merging of MLLMs.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC