Do Audio-Visual Segmentation Models Truly Segment Sounding Objects?
Journal:
arXiv
Published Date:
Feb 1, 2025
Abstract
Unlike traditional visual segmentation, audio-visual segmentation (AVS)
requires the model not only to identify and segment objects but also to
determine whether they are sound sources. Recent AVS approaches, leveraging
transformer architectures and powerful foundation models like SAM, have
achieved impressive performance on standard benchmarks. Yet, an important
question remains: Do these models genuinely integrate audio-visual cues to
segment sounding objects? In this paper, we systematically investigate this
issue in the context of robust AVS. Our study reveals a fundamental bias in
current methods: they tend to generate segmentation masks based predominantly
on visual salience, irrespective of the audio context. This bias results in
unreliable predictions when sounds are absent or irrelevant. To address this
challenge, we introduce AVSBench-Robust, a comprehensive benchmark
incorporating diverse negative audio scenarios including silence, ambient
noise, and off-screen sounds. We also propose a simple yet effective approach
combining balanced training with negative samples and classifier-guided
similarity learning. Our extensive experiments show that state-of-theart AVS
methods consistently fail under negative audio conditions, demonstrating the
prevalence of visual bias. In contrast, our approach achieves remarkable
improvements in both standard metrics and robustness measures, maintaining
near-perfect false positive rates while preserving highquality segmentation
performance.
