Recent developments in computer graphics, hardware, artificial intelligence (AI), and human-computer interaction likely lead to extended reality (XR) devices and setups being more pervasive. While these devices and setups provide users with interactive, engaging, and immersive experiences with different sensing modalities, such as eye and hand trackers, many non-player characters are utilized in a pre-scripted way or by conventional AI techniques. In this paper, we argue for using large language models (LLMs) in XR by embedding them in virtual avatars or as narratives to facilitate more inclusive experiences through prompt engineering according to user profiles and fine-tuning the LLMs for particular purposes. We argue that such inclusion will facilitate diversity for XR use. In addition, we believe that with the versatile conversational capabilities of LLMs, users will engage more with XR environments, which might help XR be more used in everyday life. Lastly, we speculate that combining the information provided to LLM-powered environments by the users and the biometric data obtained through the sensors might lead to novel privacy invasions. While studying such possible privacy invasions, user privacy concerns and preferences should also be investigated. In summary, despite some challenges, embedding LLMs into XR is a promising and novel research area with several opportunities.

翻译：计算机图形学、硬件、人工智能（AI）及人机交互领域的最新进展，很可能推动扩展现实（XR）设备与系统变得更加普及。尽管这些设备和系统能为用户提供融合眼动追踪、手部追踪等多种感知模态的交互式、沉浸式体验，但其中许多非玩家角色仍采用预设脚本或传统AI技术。本文主张通过将大型语言模型（LLMs）嵌入虚拟化身或叙事框架中，借助基于用户画像的提示工程及针对特定目标的微调，在XR中实现更具包容性的体验。我们认为这种包容性将促进XR应用的多样性。同时，我们相信LLMs强大的对话能力将增强用户与XR环境的互动，这可能推动XR在日常生活场景中的更广泛应用。最后，我们推测用户向LLM驱动的环境提供的信息与传感器获取的生物特征数据相结合，可能导致新型隐私侵犯。在探究这类潜在隐私风险时，还需同步研究用户的隐私关切与偏好。综上所述，尽管存在若干挑战，将LLMs嵌入XR仍是一个充满机遇且具有创新前景的研究领域。