Multimodal interactions have been shown to be more flexible, efficient, and adaptable for diverse users and tasks than traditional graphical interfaces. However, existing multimodal development frameworks either do not handle the complexity and compositionality of multimodal commands well or require developers to write a substantial amount of code to support these multimodal interactions. In this paper, we present ReactGenie, a programming framework that uses a shared object-oriented state abstraction to support building complex multimodal mobile applications. Having different modalities share the same state abstraction allows developers using ReactGenie to seamlessly integrate and compose these modalities to deliver multimodal interaction. ReactGenie is a natural extension to the existing workflow of building a graphical app, like the workflow with React-Redux. Developers only have to add a few annotations and examples to indicate how natural language is mapped to the user-accessible functions in the program. ReactGenie automatically handles the complex problem of understanding natural language by generating a parser that leverages large language models. We evaluated the ReactGenie framework by using it to build three demo apps. We evaluated the accuracy of the language parser using elicited commands from crowd workers and evaluated the usability of the generated multimodal app with 16 participants. Our results show that ReactGenie can be used to build versatile multimodal applications with highly accurate language parsers, and the multimodal app can lower users' cognitive load and task completion time.

翻译：多模态交互已被证明比传统图形界面更灵活、高效且能适应多样化用户与任务需求。然而，现有多模态开发框架要么难以有效处理多模态命令的复杂性与组合性，要么需要开发者编写大量代码以支持此类交互。本文提出ReactGenie编程框架，该框架通过共享面向对象状态抽象，支持构建复杂多模态移动应用。使不同模态共享同一状态抽象，可令使用ReactGenie的开发者无缝集成与组合多种模态，从而实现多模态交互。ReactGenie是对现有图形应用开发流程（如React-Redux工作流）的自然扩展。开发者仅需添加少量注解和示例，即可指明自然语言与程序用户可访问函数之间的映射关系。ReactGenie通过生成依赖大型语言模型的解析器，自动处理自然语言理解这一复杂问题。我们通过构建三个演示应用对ReactGenie框架进行了评估，利用众包工作者激发式命令测试语言解析器精度，并召集16名参与者评估生成的多模态应用可用性。结果表明：ReactGenie可构建具备高精度语言解析器的通用型多模态应用，且其多模态应用能降低用户认知负荷与任务完成时间。