TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion
Journal:
arXiv
Published Date:
May 6, 2025
Abstract
Video frame interpolation (VFI) that leverages the bio-inspired event cameras
as guidance has recently shown better performance and memory efficiency than
the frame-based methods, thanks to the event cameras' advantages, such as high
temporal resolution. A hurdle for event-based VFI is how to effectively deal
with non-linear motion, caused by the dynamic changes in motion direction and
speed within the scene. Existing methods either use events to estimate sparse
optical flow or fuse events with image features to estimate dense optical flow.
Unfortunately, motion errors often degrade the VFI quality as the continuous
motion cues from events do not align with the dense spatial information of
images in the temporal dimension. In this paper, we find that object motion is
continuous in space, tracking local regions over continuous time enables more
accurate identification of spatiotemporal feature correlations. In light of
this, we propose a novel continuous point tracking-based VFI framework, named
TimeTracker. Specifically, we first design a Scene-Aware Region Segmentation
(SARS) module to divide the scene into similar patches. Then, a Continuous
Trajectory guided Motion Estimation (CTME) module is proposed to track the
continuous motion trajectory of each patch through events. Finally,
intermediate frames at any given time are generated through global motion
optimization and frame refinement. Moreover, we collect a real-world dataset
that features fast non-linear motion. Extensive experiments show that our
method outperforms prior arts in both motion estimation and frame interpolation
quality.
