We propose a speaker-attributed (SA) Whisper-based model for multi-talker speech recognition that combines target-speaker modeling with serialized output training (SOT). Our approach leverages a Diarization-Conditioned Whisper (DiCoW) encoder to extract target-speaker embeddings, which are concatenated into a single representation and passed to a shared decoder. This enables the model to transcribe overlapping speech as a serialized output stream with speaker tags and timestamps. In contrast to target-speaker ASR systems such as DiCoW, which decode each speaker separately, our approach performs joint decoding, allowing the decoder to condition on the context of all speakers simultaneously. Experiments show that the model outperforms existing SOT-based approaches and surpasses DiCoW on multi-talker mixtures (e.g., LibriMix).

翻译：本文提出一种基于Whisper的说话人归属多说话人语音识别模型，该模型将目标说话人建模与序列化输出训练相结合。我们的方法利用说话人日志条件化Whisper编码器提取目标说话人嵌入向量，将其拼接为单一表征后输入共享解码器。这使得模型能够将重叠语音转写为带有说话人标签和时间戳的序列化输出流。与DiCoW等需分别解码各说话人的目标说话人语音识别系统不同，我们的方法采用联合解码机制，使解码器能够同时基于所有说话人的上下文信息进行条件化生成。实验表明，该模型在多项指标上优于现有基于序列化输出训练的方法，并在多说话人混合语音数据集（如LibriMix）上超越了DiCoW系统的性能。