We tackle the challenge of building real-world multimodal assistants for complex real-world tasks. We describe the practicalities and challenges of developing and deploying GRILLBot, a leading (first and second prize winning in 2022 and 2023) system deployed in the Alexa Prize TaskBot Challenge. Building on our Open Assistant Toolkit (OAT) framework, we propose a hybrid architecture that leverages Large Language Models (LLMs) and specialised models tuned for specific subtasks requiring very low latency. OAT allows us to define when, how and which LLMs should be used in a structured and deployable manner. For knowledge-grounded question answering and live task adaptations, we show that LLM reasoning abilities over task context and world knowledge outweigh latency concerns. For dialogue state management, we implement a code generation approach and show that specialised smaller models have 84% effectiveness with 100x lower latency. Overall, we provide insights and discuss tradeoffs for deploying both traditional models and LLMs to users in complex real-world multimodal environments in the Alexa TaskBot challenge. These experiences will continue to evolve as LLMs become more capable and efficient -- fundamentally reshaping OAT and future assistant architectures.

翻译：我们致力于解决为复杂现实任务构建多模态助手的挑战。本文阐述了GRILLBot系统（在Alexa Prize TaskBot挑战赛中分别于2022年和2023年荣获一等奖和二等奖的领先系统）开发与部署中的实际问题和挑战。基于我们开源的Open Assistant Toolkit (OAT)框架，我们提出了一种混合架构，该架构结合了大语言模型（LLM）与为特定低延迟子任务调优的专用模型。OAT使我们能够以结构化且可部署的方式定义何时、如何以及使用哪些LLM。对于基于知识的问答和实时任务适配，我们证明LLM在任务上下文和世界知识上的推理能力足以抵消延迟问题。对于对话状态管理，我们实现了代码生成方法，并证明专用小型模型在延迟降低100倍的情况下仍能达到84%的有效性。总体而言，我们提供了在Alexa TaskBot挑战赛复杂现实多模态环境中部署传统模型与LLM的见解和权衡讨论。随着LLM能力的增强与效率的提升，这些经验将不断演进——从根本上重塑OAT及未来助手的架构。