Annotation of multimedia data by humans is time-consuming and costly, while reliable automatic generation of semantic metadata is a major challenge. We propose a framework to extract semantic metadata from automatically generated video captions. As metadata, we consider entities, the entities' properties, relations between entities, and the video category. We employ two state-of-the-art dense video captioning models with masked transformer (MT) and parallel decoding (PVDC) to generate captions for videos of the ActivityNet Captions dataset. Our experiments show that it is possible to extract entities, their properties, relations between entities, and the video category from the generated captions. We observe that the quality of the extracted information is mainly influenced by the quality of the event localization in the video as well as the performance of the event caption generation.

翻译：人类对多模态数据的标注耗时且成本高昂，而自动生成可靠语义元数据仍是一项重大挑战。我们提出一个框架，用于从自动生成的视频字幕中提取语义元数据。作为元数据，我们考虑实体、实体属性、实体间关系以及视频类别。采用两种先进的密集视频字幕生成模型——基于掩码Transformer（MT）和并行解码（PVDC）——为ActivityNet Captions数据集中的视频生成字幕。实验表明，从生成的字幕中提取实体、实体属性、实体间关系及视频类别是可行的。我们观察到，提取信息的质量主要受视频中事件定位质量以及事件字幕生成性能的影响。