Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning
Journal:
arXiv
Published Date:
May 26, 2025
Abstract
Audio-visual zero-shot learning (ZSL) has been extensively researched for its
capability to classify video data from unseen classes during training.
Nevertheless, current methodologies often struggle with background scene biases
and inadequate motion detail. This paper proposes a novel dual-stream
Multi-Timescale Motion-Decoupled Spiking Transformer (MDST++), which decouples
contextual semantic information and sparse dynamic motion information. The
recurrent joint learning unit is proposed to extract contextual semantic
information and capture joint knowledge across various modalities to understand
the environment of actions. By converting RGB images to events, our method
captures motion information more accurately and mitigates background scene
biases. Moreover, we introduce a discrepancy analysis block to model audio
motion information. To enhance the robustness of SNNs in extracting temporal
and motion cues, we dynamically adjust the threshold of Leaky
Integrate-and-Fire neurons based on global motion and contextual semantic
information. Our experiments validate the effectiveness of MDST++,
demonstrating their consistent superiority over state-of-the-art methods on
mainstream benchmarks. Additionally, incorporating motion and multi-timescale
information significantly improves HM and ZSL accuracy by 26.2\% and 39.9\%.
