Reinforcement learning from human or AI feedback (RLHF/RLAIF) for speech-in/speech-out dialogue systems (SDS) remains underexplored, with prior work largely limited to single semantic rewards applied at the utterance level. Such setups overlook the multi-dimensional and multi-modal nature of conversational quality, which encompasses semantic coherence, audio naturalness, speaker consistency, emotion alignment, and turn-taking behavior. Moreover, they are fundamentally mismatched with duplex spoken dialogue systems that generate responses incrementally, where agents must make decisions based on partial utterances. We address these limitations with the first multi-reward RLAIF framework for SDS, combining semantic, audio-quality, and emotion-consistency rewards. To align utterance-level preferences with incremental, blockwise decoding in duplex models, we apply turn-level preference sampling and aggregate per-block log-probabilities within a single DPO objective. We present the first systematic study of preference learning for improving SDS quality in both multi-turn Chain-of-Thought and blockwise duplex models, and release a multi-reward DPO dataset to support reproducible research. Experiments show that single-reward RLAIF selectively improves its targeted metric, while joint multi-reward training yields consistent gains across semantic quality and audio naturalness. These results highlight the importance of holistic, multi-reward alignment for practical conversational SDS.

翻译：针对语音输入/语音输出对话系统（SDS）的人类或AI反馈强化学习（RLHF/RLAIF）研究仍显不足，先前工作主要局限于在话语层面应用单一语义奖励。此类设置忽视了会话质量的多维性与多模态特性，包括语义连贯性、音频自然度、说话人一致性、情感对齐及话轮转换行为。此外，这些方法与生成增量响应双工语音对话系统的运行机制存在根本性错配——此类系统需基于局部话语片段进行决策。我们通过首个面向SDS的多奖励RLAIF框架解决了这些局限，该框架融合了语义奖励、音频质量奖励与情感一致性奖励。为使话语级偏好与双工模型的增量分块解码机制对齐，我们采用话轮级偏好采样策略，并在单一DPO目标函数内聚合分块对数概率。本研究首次系统探讨了通过偏好学习提升SDS质量的方案，涵盖多轮思维链模型与分块双工模型，并发布了多奖励DPO数据集以支持可复现研究。实验表明：单奖励RLAIF能针对性提升其目标指标，而联合多奖励训练可在语义质量与音频自然度方面实现持续增益。这些结果凸显了面向实用会话型SDS进行整体性多奖励对齐的重要性。