Large Language Models (LLMs) have showcased remarkable capabilities surpassing conventional NLP challenges, creating opportunities for use in production use cases. Towards this goal, there is a notable shift to building compound AI systems, wherein LLMs are integrated into an expansive software infrastructure with many components like models, retrievers, databases and tools. In this paper, we introduce a blueprint architecture for compound AI systems to operate in enterprise settings cost-effectively and feasibly. Our proposed architecture aims for seamless integration with existing compute and data infrastructure, with ``stream'' serving as the key orchestration concept to coordinate data and instructions among agents and other components. Task and data planners, respectively, break down, map, and optimize tasks and data to available agents and data sources defined in respective registries, given production constraints such as accuracy and latency.

翻译：大型语言模型（LLM）已展现出超越传统自然语言处理挑战的卓越能力，为生产环境中的实际应用创造了机遇。为实现这一目标，当前正显著转向构建复合人工智能系统，即将LLM集成到包含模型、检索器、数据库与工具等多组件的扩展软件基础设施中。本文提出一种适用于企业环境、兼具成本效益与可行性的复合人工智能系统蓝图架构。该架构旨在实现与现有计算及数据基础设施的无缝集成，其中“流”作为核心编排概念，用于协调智能体及其他组件间的数据与指令。任务规划器与数据规划器分别在给定准确性与延迟等生产约束条件下，将任务与数据分解、映射并优化至各自注册表中定义的可用智能体及数据源。