By combining voice and touch interactions, multimodal interfaces can surpass the efficiency of either modality alone. This paper targets complex interactions, where users can issue multimodal commands that translate into one of the possible exponential combinations of actions/function invocations. This paper presents ReactGenie, a programming framework where developers can code with simple object-oriented abstractions and labeled user-invocable primitives. ReactGenie translates multimodal user commands into ReactGenieDSL, a domain-specific language we created for this purpose, using a neural semantic parser based on large-language models. The ReactGenie runtime interprets the parsed ReactGenieDSL and composes primitives to implement complex user commands. As a result, ReactGenie provides an unprecedented level of richness in user interactions. Our evaluation showed that 12 developers can learn and build a ReactGenie application in under 2.5 hours on average. In addition, compared with a traditional GUI, end users can complete tasks faster and with less task load using ReactGenie apps.

翻译：通过融合语音与触控交互，多模态界面能够超越单一模态的效率。本文聚焦于复杂交互场景——用户可发出多模态指令，这些指令需转化为指数级可能的操作/函数调用组合。我们提出ReactGenie编程框架，开发者可借助简单的面向对象抽象与带标签的用户可调用原语进行编码。该框架利用基于大型语言模型的神经语义解析器，将用户多模态指令转化为专门设计的领域特定语言ReactGenieDSL。ReactGenie运行时环境对解析后的DSL进行解释，并通过组合原语实现复杂用户指令。因此，ReactGenie提供了前所未有的用户交互丰富度。评估表明，12名开发者平均在2.5小时内即可学会并构建ReactGenie应用。此外，与传统图形界面相比，终端用户使用ReactGenie应用完成任务速度更快、负荷更小。