Speaker extraction and diarization are two enabling techniques for real-world speech applications. Speaker extraction aims to extract a target speaker's voice from a speech mixture, while speaker diarization demarcates speech segments by speaker, annotating `who spoke when'. Previous studies have typically treated the two tasks independently. In practical applications, it is more meaningful to have knowledge about `who spoke what and when', which is captured by the two tasks. The two tasks share a similar objective of disentangling speakers. Speaker extraction operates in the frequency domain, whereas diarization is in the temporal domain. It is logical to believe that speaker activities obtained from speaker diarization can benefit speaker extraction, while the extracted speech offers more accurate speaker activity detection than the speech mixture. In this paper, we propose a unified model called Universal Speaker Extraction and Diarization (USED) to address output inconsistency and scenario mismatch issues. It is designed to manage speech mixture with varying overlap ratios and variable number of speakers. We show that the USED model significantly outperforms the competitive baselines for speaker extraction and diarization tasks on LibriMix and SparseLibriMix datasets. We further validate the diarization performance on CALLHOME, a dataset based on real recordings, and experimental results indicate that our model surpasses recently proposed approaches.

翻译：摘要：说话人提取与说话人日志化是实际语音应用中的两项关键技术。说话人提取旨在从混合语音中提取目标说话人的声音，而说话人日志化则按说话人划分语音片段，标注"谁在何时说话"。以往研究通常将这两项任务独立处理。在实际应用中，获取"谁说了什么以及何时说的"这一由两项任务共同捕捉的信息更具意义。这两项任务共享相似的目标——解耦说话人。说话人提取在频域中操作，而说话人日志化则在时域中进行。逻辑上认为，从说话人日志化获得的说话人活动信息可促进说话人提取，同时，提取出的语音相较于混合语音能提供更准确的说话人活动检测。本文提出一种名为通用说话人提取与说话人日志化（USED）的统一模型，以解决输出不一致性与场景不匹配问题。该模型设计用于处理具有不同重叠比率和可变说话人数的混合语音。我们证明，USED模型在LibriMix和SparseLibriMix数据集上的说话人提取与说话人日志化任务中显著优于竞争基线。我们进一步在基于真实录音的数据集CALLHOME上验证了日志化性能，实验结果表明，我们的模型超越了近期提出的方法。