This study proposes FlexiVoice, a text-to-speech (TTS) synthesis system capable of flexible style control with zero-shot voice cloning. The speaking style is controlled by a natural-language instruction and the voice timbre is provided by a speech reference in zero-shot manner. FlexiVoice is built with an LLM core, which takes text as input, and also takes an optional natural language instruction and an optional speech reference to control style and timbre, respectively. FlexiVoice is equipped with a novel Progressive Post-Training (PPT) scheme that progressively unlocks accurate and flexible controllability. In particular, it first employs Direct Preference Optimization (DPO) to enable FlexiVoice to accurately follow both natural language instruction and speech reference simultaneously. It then uses a multi-objective Group Relative Policy Optimization (GRPO) to disentangle style instruction, reference timbre, and textual content. Finally, it adapts instruction GRPO for more advanced instruction following. Experimental results show that FlexiVoice surpasses competing baselines and demonstrates strong capability in decoupling control factors. Human evaluations further confirm its naturalness, controllability, and robustness. Audio samples are available at https://flexi-voice.github.io.

翻译：本研究提出FlexiVoice，一种能够通过零样本语音克隆实现灵活风格控制的文本转语音（TTS）合成系统。其说话风格由自然语言指令控制，而音色则通过零样本方式由语音参考提供。FlexiVoice以大型语言模型（LLM）为核心构建，该模型以文本作为输入，并可选择性地接收自然语言指令和语音参考，分别用于控制风格和音色。FlexiVoice采用了一种新颖的渐进式后训练（PPT）方案，逐步解锁准确且灵活的可控性。具体而言，首先采用直接偏好优化（DPO）使FlexiVoice能够同时准确遵循自然语言指令和语音参考。随后，使用多目标组相对策略优化（GRPO）来解耦风格指令、参考音色与文本内容。最后，通过指令GRPO的适配实现更高级的指令跟随能力。实验结果表明，FlexiVoice超越了现有基线模型，并展现出强大的控制因子解耦能力。人工评估进一步证实了其自然度、可控性和鲁棒性。音频样本可在 https://flexi-voice.github.io 获取。