Human conversational styles are measured by the sense of humor, personality, and tone of voice. These characteristics have become essential for conversational intelligent virtual assistants. However, most of the state-of-the-art intelligent virtual assistants (IVAs) are failed to interpret the affective semantics of human voices. This research proposes an anthropomorphic intelligent system that can hold a proper human-like conversation with emotion and personality. A voice style transfer method is also proposed to map the attributes of a specific emotion. Initially, the frequency domain data (Mel-Spectrogram) is created by converting the temporal audio wave data, which comprises discrete patterns for audio features such as notes, pitch, rhythm, and melody. A collateral CNN-Transformer-Encoder is used to predict seven different affective states from voice. The voice is also fed parallelly to the deep-speech, an RNN model that generates the text transcription from the spectrogram. Then the transcripted text is transferred to the multi-domain conversation agent using blended skill talk, transformer-based retrieve-and-generate generation strategy, and beam-search decoding, and an appropriate textual response is generated. The system learns an invertible mapping of data to a latent space that can be manipulated and generates a Mel-spectrogram frame based on previous Mel-spectrogram frames to voice synthesize and style transfer. Finally, the waveform is generated using WaveGlow from the spectrogram. The outcomes of the studies we conducted on individual models were auspicious. Furthermore, users who interacted with the system provided positive feedback, demonstrating the system's effectiveness.

翻译：人类对话风格通过幽默感、个性和语调来衡量。这些特征对于对话式智能虚拟助手至关重要。然而，当前最先进的智能虚拟助手（IVAs）大多无法解释人类语音中的情感语义。本研究提出了一种具备情感和个性的拟人化智能系统，能够进行类似人类的自然对话。此外，还提出了一种语音风格迁移方法，用于映射特定情感属性。首先，通过将时域音频数据转换为频域数据（梅尔频谱图），生成包含音符、音高、节奏和旋律等音频特征的离散模式。随后，使用并联的CNN-Transformer-Encoder从语音中预测七种不同的情感状态。同时，语音被并行输入至DeepSpeech（一种基于RNN的模型），该模型从频谱图中生成文本转录。接着，转录文本通过混合技巧对话（Blended Skill Talk）、基于Transformer的检索-生成策略和束搜索解码（Beam-Search Decoding）传输至多领域对话代理，并生成合适的文本响应。该系统学习数据到潜在空间的可逆映射，该空间可被操控，并基于先前的梅尔频谱图帧生成新的梅尔频谱图帧，用于语音合成和风格迁移。最后，使用WaveGlow从频谱图中生成波形。我们对各单独模型进行的研究结果均显示出良好前景。此外，与系统交互的用户提供了积极反馈，验证了系统的有效性。