The speech-to-singing (STS) voice conversion task aims to generate singing samples corresponding to speech recordings while facing a major challenge: the alignment between the target (singing) pitch contour and the source (speech) content is difficult to learn in a text-free situation. This paper proposes AlignSTS, an STS model based on explicit cross-modal alignment, which views speech variance such as pitch and content as different modalities. Inspired by the mechanism of how humans will sing the lyrics to the melody, AlignSTS: 1) adopts a novel rhythm adaptor to predict the target rhythm representation to bridge the modality gap between content and pitch, where the rhythm representation is computed in a simple yet effective way and is quantized into a discrete space; and 2) uses the predicted rhythm representation to re-align the content based on cross-attention and conducts a cross-modal fusion for re-synthesize. Extensive experiments show that AlignSTS achieves superior performance in terms of both objective and subjective metrics. Audio samples are available at https://alignsts.github.io.

翻译：语音到歌声（STS）转换任务旨在根据语音录音生成对应的歌声样本，其主要挑战在于：在无文本情境下，目标（歌声）音高轮廓与源（语音）内容之间的对齐关系难以学习。本文提出AlignSTS——一种基于显式跨模态对齐的STS模型，该模型将音高、内容等语音变量视为不同模态。受人类如何将歌词唱成旋律的机制启发，AlignSTS：1）采用新型节奏适配器预测目标节奏表征，以弥合内容与音高之间的模态差距——该节奏表征以简洁有效的方式计算，并量化至离散空间；2）利用预测的节奏表征通过交叉注意力机制重新对齐内容，并通过跨模态融合实现语音重建。大量实验表明，AlignSTS在客观与主观指标上均取得了优越性能。音频样本详见https://alignsts.github.io。