Exploring Temporally-Aware Features for Point Tracking
Journal:
arXiv
Published Date:
Jan 21, 2025
Abstract
Point tracking in videos is a fundamental task with applications in robotics,
video editing, and more. While many vision tasks benefit from pre-trained
feature backbones to improve generalizability, point tracking has primarily
relied on simpler backbones trained from scratch on synthetic data, which may
limit robustness in real-world scenarios. Additionally, point tracking requires
temporal awareness to ensure coherence across frames, but using
temporally-aware features is still underexplored. Most current methods often
employ a two-stage process: an initial coarse prediction followed by a
refinement stage to inject temporal information and correct errors from the
coarse stage. These approach, however, is computationally expensive and
potentially redundant if the feature backbone itself captures sufficient
temporal information.
In this work, we introduce Chrono, a feature backbone specifically designed
for point tracking with built-in temporal awareness. Leveraging pre-trained
representations from self-supervised learner DINOv2 and enhanced with a
temporal adapter, Chrono effectively captures long-term temporal context,
enabling precise prediction even without the refinement stage. Experimental
results demonstrate that Chrono achieves state-of-the-art performance in a
refiner-free setting on the TAP-Vid-DAVIS and TAP-Vid-Kinetics datasets, among
common feature backbones used in point tracking as well as DINOv2, with
exceptional efficiency. Project page: https://cvlab-kaist.github.io/Chrono/
