This study investigates innovative interaction designs for communication and collaborative learning between learners of mixed hearing and signing abilities, leveraging advancements in mixed reality technologies like Apple Vision Pro and generative AI for animated avatars. Adopting a participatory design approach, we engaged 15 d/Deaf and hard of hearing (DHH) students to brainstorm ideas for an AI avatar with interpreting ability (sign language to English, voice to English) that would facilitate their face-to-face communication with hearing peers. Participants envisioned the AI avatars to address some issues with human interpreters, such as lack of availability, and provide affordable options to expensive personalized interpreting service. Our findings indicate a range of preferences for integrating the AI avatars with actual human figures of both DHH and hearing communication partners. The participants highlighted the importance of having control over customizing the AI avatar, such as AI-generated signs, voices, facial expressions, and their synchronization for enhanced emotional display in communication. Based on our findings, we propose a suite of design recommendations that balance respecting sign language norms with adherence to hearing social norms. Our study offers insights on improving the authenticity of generative AI in scenarios involving specific, and sometimes unfamiliar, social norms.

翻译：本研究探讨了利用Apple Vision Pro等混合现实技术及生成式AI动画虚拟形象的进展，为不同听力和手语能力的学习者之间的交流与协作学习设计创新交互方案。通过采用参与式设计方法，我们邀请了15名聋人及重听学生共同构思具备口译能力（手语转英语、语音转英语）的AI虚拟形象，以促进他们与健听同伴的面对面交流。参与者期望AI虚拟形象能够解决人工口译员存在的某些问题，例如人员短缺，并为昂贵的个性化口译服务提供经济替代方案。研究发现，参与者对AI虚拟形象与聋听双方真实人像的融合方式存在多元偏好。他们特别强调了对AI虚拟形象定制功能的控制权的重要性，包括AI生成的手语动作、语音、面部表情及其同步协调，以增强交流中的情感表达。基于研究结果，我们提出了一套设计建议，旨在平衡对手语规范的尊重与对健听社会习俗的遵循。本研究为提升生成式AI在特定（有时是陌生的）社会规范场景中的真实性提供了见解。