TY - JOUR
T1 - Language Interprets Vision
T2 - Adaptive Encoding and Decoding for Referring Image Segmentation
AU - Qi, A.
AU - Zhao, Sanyuan
AU - Dong, Xingping
AU - Shen, Jianbing
N1 - Publisher Copyright:
© 2024 Tsinghua University Press.
PY - 2026
Y1 - 2026
N2 - Referring image segmentation aims to segment the referent with natural linguistic expressions. Due to the distinct modality properties of the image and language, it is challenging to effectively align token embeddings with visual regions. Different from existing methods of coordinate linguistics for the specific visual region, we propose a novel referring image segmentation paradigm, language interprets vision (LIV), which densely fine-grained aligns the visual and linguistic modalities, and fuse the multi-modal biases effectively. LIV resorts to re-encoding visual features on compositional dimensions of , which interprets vision through linguistic expression and makes cross-modality alignment denser. More specifically, we innovatively consider the adjacency of visual regions on the channel level to promote channel semantic consistency and propagate fine-grained semantics in the whole segmentation procedure. In addition, we also theoretically analyze that LIV effectively enriches the representation space and makes the comprehensive modality-fused biases more generalized, which boosts the precision of mask prediction. Extensive experimental results on three benchmarks validate that our proposed framework significantly outperforms other methods by a remarkable margin.
AB - Referring image segmentation aims to segment the referent with natural linguistic expressions. Due to the distinct modality properties of the image and language, it is challenging to effectively align token embeddings with visual regions. Different from existing methods of coordinate linguistics for the specific visual region, we propose a novel referring image segmentation paradigm, language interprets vision (LIV), which densely fine-grained aligns the visual and linguistic modalities, and fuse the multi-modal biases effectively. LIV resorts to re-encoding visual features on compositional dimensions of , which interprets vision through linguistic expression and makes cross-modality alignment denser. More specifically, we innovatively consider the adjacency of visual regions on the channel level to promote channel semantic consistency and propagate fine-grained semantics in the whole segmentation procedure. In addition, we also theoretically analyze that LIV effectively enriches the representation space and makes the comprehensive modality-fused biases more generalized, which boosts the precision of mask prediction. Extensive experimental results on three benchmarks validate that our proposed framework significantly outperforms other methods by a remarkable margin.
KW - Transformer
KW - attention
KW - cross modal
KW - referring image segmentation (RIS)
KW - segmentation
UR - https://www.scopus.com/pages/publications/105029745895
U2 - 10.26599/CVM.2025.9450427
DO - 10.26599/CVM.2025.9450427
M3 - Article
AN - SCOPUS:105029745895
SN - 2096-0433
VL - 12
SP - 189
EP - 202
JO - Computational Visual Media
JF - Computational Visual Media
IS - 1
ER -