We propose StyleCap, a method to generate natural language descriptions of speaking styles appearing in speech. Although most of conventional techniques for para-/non-linguistic information recognition focus on the category classification or the intensity estimation of pre-defined labels, they cannot provide the reasoning of the recognition result in an interpretable manner. StyleCap is a first step towards an end-to-end method for generating speaking-style prompts from speech, i.e., automatic speaking-style captioning. StyleCap is trained with paired data of speech and natural language descriptions. We train neural networks that convert a speech representation vector into prefix vectors that are fed into a large language model (LLM)-based text decoder. We explore an appropriate text decoder and speech feature representation suitable for this new task. The experimental results demonstrate that our StyleCap leveraging richer LLMs for the text decoder, speech self-supervised learning (SSL) features, and sentence rephrasing augmentation improves the accuracy and diversity of generated speaking-style captions. Samples of speaking-style captions generated by our StyleCap are publicly available.

翻译：摘要：我们提出StyleCap方法，用于生成语音中说话风格的自然语言描述。尽管传统的副语言/非语言信息识别技术主要侧重于预定义标签的类别分类或强度估计，但这些方法无法以可解释的方式提供识别结果的推理依据。StyleCap是实现从语音生成说话风格提示词（即自动说话风格描述）的端到端方法的第一步。StyleCap使用语音与自然语言描述的配对数据进行训练。我们训练神经网络将语音表征向量转换为前缀向量，并将其输入基于大语言模型（LLM）的文本解码器。我们探索了适合该新任务的文本解码器与语音特征表征方案。实验结果表明，利用更丰富的LLM作为文本解码器、语音自监督学习（SSL）特征以及句子改写增强的StyleCap方法，能够提升所生成说话风格描述的准确性与多样性。通过StyleCap生成的说话风格描述样本现已公开提供。