While large language models (LLMs) have accelerated 2D software development through intent-driven "vibe coding", prototyping intelligent Extended Reality (XR) experiences remains a major challenge. The fundamental barrier is not just the steep learning curve for human creators, but that low-level sensor APIs and complex game engine hierarchies are ill-suited for LLM reasoning, routinely exceeding context windows and inducing syntax hallucinations. To bridge this gap, we contribute XR Blocks, an open-source, LLM-native WebXR framework. Unlike traditional engines, XR Blocks introduces a semantic "Reality Model" that aligns spatial computing primitives (users, physical environments, and agents) with natural language, providing a robust, concise vocabulary optimized for generative AI. Building upon this foundation, we present Vibe Coding XR, an end-to-end prototyping workflow that leverages LLMs to translate high-level prompts (e.g., "create a dandelion that reacts to my hand") directly into functional, physics-aware mixed-reality applications. To minimize the friction of on-device testing, the workflow introduces a seamless desktop "simulated reality" to headset deployment loop. Finally, we introduce VCXR60, a pilot dataset of 60 XR prompts paired with an automated evaluation pipeline. Our technical evaluation demonstrates high one-shot execution success, enabling practitioners to bypass lowlevel hurdles and rapidly move from "idea to reality". Code and live demos are available at https://github.com/google/xrblocks and http://xrblocks.github.io/gem.
翻译:摘要:尽管大型语言模型(LLM)通过基于意图的“氛围式编程”加速了二维软件开发,但面向智能扩展现实(XR)体验的原型开发仍面临重大挑战。根本障碍不仅在于人类创作者陡峭的学习曲线,更在于低层级传感器接口与复杂游戏引擎层次结构难以适配LLM推理,常常超出上下文窗口并引发语法幻觉。为弥合这一鸿沟,我们提出了XR Blocks——一个开源、原生适配LLM的WebXR框架。与传统引擎不同,XR Blocks引入了一个语义化的“现实模型”,将空间计算基元(用户、物理环境及智能体)与自然语言对齐,为生成式人工智能提供了稳健且简洁的优化词汇表。在此基础上,我们构建了Vibe Coding XR——一个端到端的原型开发工作流,利用LLM将高级提示(例如“生成一朵响应我手势的蒲公英”)直接转化为具备物理感知的混合现实应用。为减少设备端测试的摩擦,该工作流引入了从桌面端“模拟现实”到头戴设备的无缝部署循环。最后,我们推出了VCXR60数据集(包含60个XR提示的试点数据集),并配以自动化评估流水线。技术评估表明,该方法在单次执行中具有高成功率,使开发者能够绕过低层级障碍,快速实现“从创意到现实”的转化。代码与在线演示参见:https://github.com/google/xrblocks 及 http://xrblocks.github.io/gem。