Fully Exploiting Vision Foundation Model's Profound Prior Knowledge for Generalizable RGB-Depth Driving Scene Parsing
Journal:
arXiv
Published Date:
Feb 10, 2025
Abstract
Recent vision foundation models (VFMs), typically based on Vision Transformer
(ViT), have significantly advanced numerous computer vision tasks. Despite
their success in tasks focused solely on RGB images, the potential of VFMs in
RGB-depth driving scene parsing remains largely under-explored. In this
article, we take one step toward this emerging research area by investigating a
feasible technique to fully exploit VFMs for generalizable RGB-depth driving
scene parsing. Specifically, we explore the inherent characteristics of RGB and
depth data, thereby presenting a Heterogeneous Feature Integration Transformer
(HFIT). This network enables the efficient extraction and integration of
comprehensive heterogeneous features without re-training ViTs. Relative depth
prediction results from VFMs, used as inputs to the HFIT side adapter, overcome
the limitations of the dependence on depth maps. Our proposed HFIT demonstrates
superior performance compared to all other traditional single-modal and
data-fusion scene parsing networks, pre-trained VFMs, and ViT adapters on the
Cityscapes and KITTI Semantics datasets. We believe this novel strategy paves
the way for future innovations in VFM-based data-fusion techniques for driving
scene parsing. Our source code is publicly available at
https://mias.group/HFIT.
