The short-form videos have explosive popularity and have dominated the new social media trends. Prevailing short-video platforms,~\textit{e.g.}, Kuaishou (Kwai), TikTok, Instagram Reels, and YouTube Shorts, have changed the way we consume and create content. For video content creation and understanding, the shot boundary detection (SBD) is one of the most essential components in various scenarios. In this work, we release a new public Short video sHot bOundary deTection dataset, named SHOT, consisting of 853 complete short videos and 11,606 shot annotations, with 2,716 high quality shot boundary annotations in 200 test videos. Leveraging this new data wealth, we propose to optimize the model design for video SBD, by conducting neural architecture search in a search space encapsulating various advanced 3D ConvNets and Transformers. Our proposed approach, named AutoShot, achieves higher F1 scores than previous state-of-the-art approaches, e.g., outperforming TransNetV2 by 4.2%, when being derived and evaluated on our newly constructed SHOT dataset. Moreover, to validate the generalizability of the AutoShot architecture, we directly evaluate it on another three public datasets: ClipShots, BBC and RAI, and the F1 scores of AutoShot outperform previous state-of-the-art approaches by 1.1%, 0.9% and 1.2%, respectively. The SHOT dataset and code can be found in https://github.com/wentaozhu/AutoShot.git .
翻译:短视频已呈现出爆炸性流行趋势,并主导了新的社交媒体潮流。主流短视频平台(如快手、TikTok、Instagram Reels和YouTube Shorts)彻底改变了我们消费和创作内容的方式。在视频内容创作与理解过程中,镜头边界检测(SBD)是各类场景中最核心的技术之一。本文发布了一个全新的公共短视频镜头边界检测数据集SHOT,包含853个完整短视频和11,606条镜头标注,其中200个测试视频包含2,716条高质量镜头边界标注。借助这一新数据资源,我们提出通过在一个封装了多种先进3D卷积网络和Transformer的搜索空间中进行神经架构搜索,以优化视频SBD的模型设计。我们的方法命名为AutoShot,在自建的SHOT数据集上进行训练和评估时,其F1分数比之前最先进的方法(如TransNetV2)高出4.2%。此外,为验证AutoShot架构的泛化能力,我们将其直接应用于另外三个公开数据集:ClipShots、BBC和RAI,其F1分数分别比之前最先进的方法高出1.1%、0.9%和1.2%。SHOT数据集和代码可在https://github.com/wentaozhu/AutoShot.git 获取。