We present a novel model for Tracking Any Point (TAP) that effectively tracks any queried point on any physical surface throughout a video sequence. Our approach employs two stages: (1) a matching stage, which independently locates a suitable candidate point match for the query point on every other frame, and (2) a refinement stage, which updates both the trajectory and query features based on local correlations. The resulting model surpasses all baseline methods by a significant margin on the TAP-Vid benchmark, as demonstrated by an approximate 20% absolute average Jaccard (AJ) improvement on DAVIS. Our model facilitates fast inference on long and high-resolution video sequences. On a modern GPU, our implementation has the capacity to track points faster than real-time, and can be flexibly extended to higher-resolution videos. Given the high-quality trajectories extracted from a large dataset, we demonstrate a proof-of-concept diffusion model which generates trajectories from static images, enabling plausible animations. Visualizations, source code, and pretrained models can be found on our project webpage.

翻译：我们提出了一种针对任意点追踪（TAP）任务的新型模型，该模型能够有效追踪视频序列中任意物理表面上的任意查询点。本方法采用两阶段架构：（1）匹配阶段——独立定位每一帧中与查询点对应的候选匹配点；（2）细化阶段——基于局部相关性更新轨迹与查询特征。在TAP-Vid基准测试中，该模型以显著优势超越所有基线方法，在DAVIS数据集上实现了约20%的绝对平均交并比（AJ）提升。模型支持长时高分辨率视频序列的快速推理，在现代GPU上可实现超实时追踪，并可灵活扩展到更高分辨率视频。基于从大规模数据集中提取的高质量轨迹，我们进一步展示了概念验证性扩散模型——该模型能从静态图像生成轨迹，从而支持合理的动画生成。可视化结果、源代码及预训练模型详见项目网页。