STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding
Journal:
arXiv
Published Date:
Mar 20, 2025
Abstract
Pre-trained on tremendous image-text pairs, vision-language models like CLIP
have demonstrated promising zero-shot generalization across numerous
image-based tasks. However, extending these capabilities to video tasks remains
challenging due to limited labeled video data and high training costs. Recent
video prompting methods attempt to adapt CLIP for video tasks by introducing
learnable prompts, but they typically rely on a single static prompt for all
video sequences, overlooking the diverse temporal dynamics and spatial
variations that exist across frames. This limitation significantly hinders the
model's ability to capture essential temporal information for effective video
understanding. To address this, we propose an integrated Spatial-TempOral
dynamic Prompting (STOP) model which consists of two complementary modules, the
intra-frame spatial prompting and inter-frame temporal prompting. Our
intra-frame spatial prompts are designed to adaptively highlight discriminative
regions within each frame by leveraging intra-frame attention and temporal
variation, allowing the model to focus on areas with substantial temporal
dynamics and capture fine-grained spatial details. Additionally, to highlight
the varying importance of frames for video understanding, we further introduce
inter-frame temporal prompts, dynamically inserting prompts between frames with
high temporal variance as measured by frame similarity. This enables the model
to prioritize key frames and enhances its capacity to understand temporal
dependencies across sequences. Extensive experiments on various video
benchmarks demonstrate that STOP consistently achieves superior performance
against state-of-the-art methods. The code is available at
https://github.com/zhoujiahuan1991/CVPR2025-STOP.
