This technical report describes our QuAVF@NTU-NVIDIA submission to the Ego4D Talking to Me (TTM) Challenge 2023. Based on the observation from the TTM task and the provided dataset, we propose to use two separate models to process the input videos and audio. By doing so, we can utilize all the labeled training data, including those without bounding box labels. Furthermore, we leverage the face quality score from a facial landmark prediction model for filtering noisy face input data. The face quality score is also employed in our proposed quality-aware fusion for integrating the results from two branches. With the simple architecture design, our model achieves 67.4% mean average precision (mAP) on the test set, which ranks first on the leaderboard and outperforms the baseline method by a large margin. Code is available at: https://github.com/hsi-che-lin/Ego4D-QuAVF-TTM-CVPR23

翻译：本技术报告介绍了我们提交至Ego4D“与我对话”（TTM）挑战赛2023的QuAVF@NTU-NVIDIA方案。基于对TTM任务及所提供数据集的观察，我们提出使用两个独立模型分别处理输入视频和音频。通过这一设计，我们得以利用所有带标注的训练数据，包括那些不包含边界框标签的数据。此外，我们借助面部关键点预测模型产生的面部质量分数，对含噪声的面部输入数据进行过滤。该面部质量分数还被用于我们提出的质量感知融合机制中，以整合两个分支的结果。凭借简洁的架构设计，我们的模型在测试集上取得了67.4%的平均精度（mAP），位列排行榜首位，并以显著优势超越了基线方法。代码开源地址：https://github.com/hsi-che-lin/Ego4D-QuAVF-TTM-CVPR23