We tackle the challenge of building real-world multimodal assistants for complex real-world tasks. We describe the practicalities and challenges of developing and deploying GRILLBot, a leading (first and second prize winning in 2022 and 2023) system deployed in the Alexa Prize TaskBot Challenge. Building on our Open Assistant Toolkit (OAT) framework, we propose a hybrid architecture that leverages Large Language Models (LLMs) and specialised models tuned for specific subtasks requiring very low latency. OAT allows us to define when, how and which LLMs should be used in a structured and deployable manner. For knowledge-grounded question answering and live task adaptations, we show that LLM reasoning abilities over task context and world knowledge outweigh latency concerns. For dialogue state management, we implement a code generation approach and show that specialised smaller models have 84% effectiveness with 100x lower latency. Overall, we provide insights and discuss tradeoffs for deploying both traditional models and LLMs to users in complex real-world multimodal environments in the Alexa TaskBot challenge. These experiences will continue to evolve as LLMs become more capable and efficient -- fundamentally reshaping OAT and future assistant architectures.

翻译：我们致力于解决为复杂现实任务构建实用多模态助手的挑战。本文阐述了在Alexa Prize TaskBot挑战赛中开发并部署领先系统GRILLBot（2022年与2023年分别获得一等奖和二等奖）的实践细节与挑战。基于我们提出的开放助手工具包（OAT）框架，我们设计了一种混合架构，该架构结合了大型语言模型（LLMs）与针对特定低延迟子任务优化的专用模型。OAT使我们能够以结构化且可部署的方式，明确界定何时、如何以及使用何种LLMs。对于知识增强的问答和实时任务适应，我们证明LLMs在任务上下文和世界知识上的推理能力优于对延迟的考量。在对话状态管理方面，我们采用代码生成方法，并证明专用小型模型在保持84%有效性的同时，延迟降低了100倍。总体而言，我们通过Alexa TaskBot挑战赛，分享了在复杂现实多模态环境中向用户部署传统模型与LLMs的实践见解与权衡讨论。随着LLMs能力与效率的持续提升——这将从根本上重塑OAT及未来助手架构——相关经验也将持续演进。