Compound AI is a distributed intelligence approach that represents a unified system orchestrating specialized AI/ML models with engineered software components into AI workflows. Compound AI production deployments must satisfy accuracy, latency, and cost objectives under varying loads. However, many deployments operate on fixed infrastructure where horizontal scaling is not viable. Existing approaches optimize solely for accuracy and do not consider changes in workload conditions. We observe that compound AI systems can switch between configurations to fit infrastructure capacity, trading accuracy for latency based on current load. This requires discovering multiple Pareto-optimal configurations from a combinatorial search space and determining when to switch between them at runtime. We present Compass, a novel framework that enables dynamic configuration switching through offline optimization and online adaptation. Compass consists of three components: COMPASS-V algorithm for configuration discovery, Planner for switching policy derivation, and Elastico Controller for runtime adaptation. COMPASS-V discovers accuracy-feasible configurations using finite-difference guided search and a combination of hill-climbing and lateral expansion. Planner profiles these configurations on target hardware and derives switching policies using a queuing theory based model. Elastico monitors queue depth and switches configurations based on derived thresholds. Across two compound AI workflows, COMPASS-V achieves 100% recall while reducing configuration evaluations by 57.5% on average compared to exhaustive search, with efficiency gains reaching 95.3% at tight accuracy thresholds. Runtime adaptation achieves 90-98% SLO compliance under dynamic load patterns, improving SLO compliance by 71.6% over static high-accuracy baselines, while simultaneously improving accuracy by 3-5% over static fast baselines.

翻译：复合AI是一种分布式智能方法，它将专用AI/ML模型与工程化软件组件编排成AI工作流，形成统一系统。复合AI生产部署需在动态负载下满足准确性、延迟和成本目标。然而，许多部署运行在固定基础设施上，无法进行水平扩展。现有方法仅针对准确性优化，未考虑工作负载条件的变化。我们观察到复合AI系统可在不同配置间切换以适配基础设施容量，基于当前负载在准确性与延迟间进行权衡。这需要从组合搜索空间中发掘多个帕累托最优配置，并在运行时确定切换时机。本文提出Compass——一种通过离线优化与在线自适应实现动态配置切换的新型框架。Compass包含三个组件：用于配置发现的COMPASS-V算法、用于切换策略推导的规划器，以及用于运行时自适应的弹性控制器。COMPASS-V采用有限差分引导搜索结合爬山算法与横向扩展方法，发现可行准确性配置。规划器在目标硬件上对这些配置进行性能剖析，并基于排队论模型推导切换策略。弹性控制器监控队列深度，根据推导阈值进行配置切换。在两个复合AI工作流实验中，COMPASS-V实现100%召回率的同时，相较于穷举搜索平均减少57.5%的配置评估次数，在严格准确性阈值下效率提升达95.3%。运行时自适应在动态负载模式下达到90-98%的服务水平目标（SLO）符合率，相较于静态高准确性基线提升71.6%的SLO符合率，同时相较于静态快速基线提升3-5%的准确性。