AdaptRGB-t: Adaptive RGB-t semantic segmentation via efficient parameter-tuning with textual guidance

TY - JOUR

T1 - AdaptRGB-t

T2 - Adaptive RGB-t semantic segmentation via efficient parameter-tuning with textual guidance

AU - Yu, Meng

AU - Yue, Yufeng

AU - Yang, Yi

AU - Fu, Mengyin

N1 - Publisher Copyright: © 2025 Elsevier B.V.

PY - 2026/2/1

Y1 - 2026/2/1

N2 - Reliable semantic segmentation is essential for intelligent systems, yet significant problems remain: 1) Existing RGB-Thermal (RGB-T) segmentation models mainly rely on visual features and lack textual information, which may lead to inaccurate segmentation when categories share similar visual characteristics. 2) While SAM excels in instance-level segmentation, integrating it with thermal images and text is hindered by modality heterogeneity and computational inefficiency. Motivated by these observations, we introduce AdaptRGB-T, a parameter-efficient fine-tuning framework using Low-Rank Adaptation (LoRA) to adapt for RGB-T semantic segmentation. Specifically, we propose an Enhanced Transformer Block (ETB) that freezes SAM's original transformer blocks and incorporates trainable LoRA layers for efficient RGB-T feature fusion. Additionally, we incorporate CLIP-generated text embeddings in the mask decoder to enable semantic alignment, which further rectifies classification errors and improves semantic understanding accuracy. Experimental results across diverse datasets demonstrate that our method achieves superior performance in challenging scenarios with fewer trainable parameters. The code will be available at https://github.com/mengyu212/AdaptRGBT.

AB - Reliable semantic segmentation is essential for intelligent systems, yet significant problems remain: 1) Existing RGB-Thermal (RGB-T) segmentation models mainly rely on visual features and lack textual information, which may lead to inaccurate segmentation when categories share similar visual characteristics. 2) While SAM excels in instance-level segmentation, integrating it with thermal images and text is hindered by modality heterogeneity and computational inefficiency. Motivated by these observations, we introduce AdaptRGB-T, a parameter-efficient fine-tuning framework using Low-Rank Adaptation (LoRA) to adapt for RGB-T semantic segmentation. Specifically, we propose an Enhanced Transformer Block (ETB) that freezes SAM's original transformer blocks and incorporates trainable LoRA layers for efficient RGB-T feature fusion. Additionally, we incorporate CLIP-generated text embeddings in the mask decoder to enable semantic alignment, which further rectifies classification errors and improves semantic understanding accuracy. Experimental results across diverse datasets demonstrate that our method achieves superior performance in challenging scenarios with fewer trainable parameters. The code will be available at https://github.com/mengyu212/AdaptRGBT.

KW - LoRA fine-tuning

KW - RGB-t semantic segmentation

KW - Segment anything model

KW - Textual guidance

UR - https://www.scopus.com/pages/publications/105022091905

U2 - 10.1016/j.neucom.2025.132060

DO - 10.1016/j.neucom.2025.132060

M3 - Article

AN - SCOPUS:105022091905

SN - 0925-2312

VL - 664

JO - Neurocomputing

JF - Neurocomputing

M1 - 132060

ER -