SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection
Journal:
arXiv
Published Date:
Dec 30, 2024
Abstract
With the rapid advancement of remote sensing technology, high-resolution
multi-modal imagery is now more widely accessible. Conventional Object
detection models are trained on a single dataset, often restricted to a
specific imaging modality and annotation format. However, such an approach
overlooks the valuable shared knowledge across multi-modalities and limits the
model's applicability in more versatile scenarios. This paper introduces a new
task called Multi-Modal Datasets and Multi-Task Object Detection (M2Det) for
remote sensing, designed to accurately detect horizontal or oriented objects
from any sensor modality. This task poses challenges due to 1) the trade-offs
involved in managing multi-modal modelling and 2) the complexities of
multi-task optimization. To address these, we establish a benchmark dataset and
propose a unified model, SM3Det (Single Model for Multi-Modal datasets and
Multi-Task object Detection). SM3Det leverages a grid-level sparse MoE backbone
to enable joint knowledge learning while preserving distinct feature
representations for different modalities. Furthermore, it integrates a
consistency and synchronization optimization strategy using dynamic learning
rate adjustment, allowing it to effectively handle varying levels of learning
difficulty across modalities and tasks. Extensive experiments demonstrate
SM3Det's effectiveness and generalizability, consistently outperforming
specialized models on individual datasets. The code is available at
https://github.com/zcablii/SM3Det.
