Toggle Main Menu Toggle Search

Open Access padlockePrints

The Newcastle University research output collection, currently available on ePrints, will shortly be moving to a new open repository platform, Figshare. To prepare for the data migration we have paused adding new content to ePrints, and will resume once the new repository is launched. During this time you will continue to have access to ePrints (but no new content will appear). We will share updates here when available.

Multi-Text Guidance Is Important: Multi-Modality Image Fusion via Large Generative Vision-Language Model

Lookup NU author(s): Dr Shidong WangORCiD

Downloads

Full text for this publication is not currently held within this repository. Alternative links are provided below where available.

Abstract

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2025. Multi-modality image fusion aims to extract complementary features from multiple source images of different modalities, generating a fused image that inherits their advantages. To address challenges in cross-modality shared feature (CMSF) extraction, single-modality specific feature (SMSF) fusion, and the absence of ground truth (GT) images, we propose MTG-Fusion, a multi-text guided model. We leverage the capabilities of large vision-language models to generate text descriptions tailored to the input images, providing novel insights for these challenges. Our model introduces a text-guided CMSF extractor (TGCE) and a text-guided SMSF fusion module (TGSF). TGCE transforms visual features into the text domain using manifold-isometric domain transform techniques and provides effective visual-text interaction based on text-vision and text-text distances. TGSF fuses each dimension of visual features with corresponding text features, creating a weight matrix utilized for SMSF fusion. We also incorporate the constructed textual GT into the loss function for collaborative training. Extensive experiments demonstrate that MTG-Fusion achieves state-of-the-art performance on infrared and visible image fusion and medical image fusion tasks. The code is available at: https://github.com/zhaolb4080/MTG-Fusion.

Publication metadata

Author(s): Wang Z, Zhao L, Zhang J, Song R, Song H, Meng J, Wang S

Publication type: Article

Publication status: Published

Journal: International Journal of Computer Vision

Year: 2025

Volume: 133

Pages: 4646-4668

Print publication date: 01/07/2025

Online publication date: 17/03/2025

Acceptance date: 24/02/2025

ISSN (print): 0920-5691

ISSN (electronic): 1573-1405

Publisher: Springer Nature

URL: https://doi.org/10.1007/s11263-025-02409-3

DOI: 10.1007/s11263-025-02409-3

Altmetrics

Altmetrics provided by Altmetric

Share