The automatic identification system (AIS) and video cameras have been widely exploited for vessel traffic surveillance in inland waterways. The AIS data could provide the vessel identity and dynamic information on vessel position and movements. In contrast, the video data could describe the visual appearances of moving vessels, but without knowing the information on identity, position and movements, etc. To further improve vessel traffic surveillance, it becomes necessary to fuse the AIS and video data to simultaneously capture the visual features, identity and dynamic information for the vessels of interest. However, traditional data fusion methods easily suffer from several potential limitations, e.g., asynchronous messages, missing data, random outliers, etc. In this work, we first extract the AIS- and video-based vessel trajectories, and then propose a deep learning-enabled asynchronous trajectory matching method (named DeepSORVF) to fuse the AIS-based vessel information with the corresponding visual targets. In addition, by combining the AIS- and video-based movement features, we also present a prior knowledge-driven anti-occlusion method to yield accurate and robust vessel tracking results under occlusion conditions. To validate the efficacy of our DeepSORVF, we have also constructed a new benchmark dataset (termed FVessel) for vessel detection, tracking, and data fusion. It consists of many videos and the corresponding AIS data collected in various weather conditions and locations. The experimental results have demonstrated that our method is capable of guaranteeing high-reliable data fusion and anti-occlusion vessel tracking.

翻译：自动识别系统（AIS）和视频摄像机已广泛应用于内河航道船舶交通监控。AIS数据可提供船舶身份及位置、运动等动态信息；而视频数据能描述运动船舶的视觉外观，但无法获知其身份、位置和运动等信息。为提升船舶交通监控性能，需融合AIS与视频数据，同时捕获目标船舶的视觉特征、身份和动态信息。然而，传统数据融合方法易面临异步消息、数据缺失、随机异常值等潜在限制。本文首先提取基于AIS和视频的船舶轨迹，继而提出一种基于深度学习的异步轨迹匹配方法（命名为DeepSORVF），将AIS船舶信息与对应的视觉目标进行融合。此外，通过结合AIS与视频的运动特征，我们还提出一种先验知识驱动的抗遮挡方法，在遮挡条件下实现精准鲁棒的船舶跟踪。为验证DeepSORVF的有效性，我们构建了新的基准数据集（命名为FVessel），用于船舶检测、跟踪及数据融合，该数据集包含多种天气条件和地点采集的多组视频及其对应AIS数据。实验结果表明，该方法能确保高可靠性的数据融合及抗遮挡船舶跟踪。