We introduce a distinctive real-time, causal, neural network-based active speaker detection system optimized for low-power edge computing. This system drives a virtual cinematography module and is deployed on a commercial device. The system uses data originating from a microphone array and a 360-degree camera. Our network requires only 127 MFLOPs per participant, for a meeting with 14 participants. Unlike previous work, we examine the error rate of our network when the computational budget is exhausted, and find that it exhibits graceful degradation, allowing the system to operate reasonably well even in this case. Departing from conventional DOA estimation approaches, our network learns to query the available acoustic data, considering the detected head locations. We train and evaluate our algorithm on a realistic meetings dataset featuring up to 14 participants in the same meeting, overlapped speech, and other challenging scenarios.

翻译：我们提出了一种独特的基于神经网络的实时因果主动说话人检测系统，专为低功耗边缘计算优化。该系统驱动虚拟摄影模块，并部署于商用设备上，使用来自麦克风阵列和360度摄像头的数据。对于14人会议，我们的网络每位参与者仅需127 MFLOPs。与先前研究不同，我们考察了计算预算耗尽时网络的错误率，发现其具有优雅降级特性，使系统在此情况下仍能合理运行。不同于传统的DOA估计方法，我们的网络学会根据检测到的头部位置查询可用声学数据。我们在包含多达14人同时参与、重叠语音及其他挑战性场景的真实会议数据集上训练并评估了该算法。