Mixed Reality (MR) platforms enable users to interact with three-dimensional holographic instructions during the assembly and fabrication of highly custom and parametric architectural constructions without the necessity of two-dimensional drawings. Previous MR fabrication projects have primarily relied on digital menus and custom buttons as the interface for user interaction with the MR environment. Despite this approach being widely adopted, it is limited in its ability to allow for direct human interaction with physical objects to modify fabrication instructions within the MR environment. This research integrates user interactions with physical objects through real-time gesture recognition as input to modify, update or generate new digital information enabling reciprocal stimuli between the physical and the virtual environment. Consequently, the digital environment is generative of the user's provided interaction with physical objects to allow seamless feedback in the fabrication process. This research investigates gesture recognition for feedback-based MR workflows for robotic fabrication, human assembly, and quality control in the construction of the UnLog Tower.

翻译：混合现实（MR）平台使用户能够在高度定制化和参数化的建筑结构组装与建造过程中，无需依赖二维图纸即可与三维全息指导信息进行交互。以往的MR建造项目主要依赖数字菜单和定制按钮作为用户与MR环境交互的界面。尽管这种方法已被广泛采用，但其在允许用户直接通过物理对象交互来修改MR环境中的建造指令方面存在局限。本研究通过实时手势识别将用户与物理对象的交互作为输入，以修改、更新或生成新的数字信息，从而实现物理环境与虚拟环境之间的双向激励。因此，数字环境能够根据用户与物理对象的交互动态生成内容，为建造过程提供无缝反馈。本研究探讨了在UnLog塔的建造过程中，基于手势识别的反馈式MR工作流在机器人建造、人工组装与质量控制中的应用。