Large Language Model based Multi-Agent Systems (LLM-MAS) have emerged as a powerful paradigm for tackling complex tasks by breaking them into collaborative workflows of specialized LLM-powered agents. However, deploying such multi-agent workloads at scale poses significant system challenges. Each user query spawns an iterative pipeline of LLM calls, greatly amplifying resource consumption compared to single-turn queries. In resource-constrained cloud settings, these workflows face non-deterministic and input-dependent costs at decode stage, heavy-tailed multi-model requirements with memory fragmentation and over-provisioning, and cross-cluster scheduling trade-offs. We present Maestro, a workload-aware scheduling system designed for LLM-MAS serving under strict GPU budgets. Maestro explicitly leverages agent semantics and roles: it predicts the output length and memory usage of each stage and uses this prediction to drive a hierarchical scheduler. At the node level, Maestro enables dynamic multi-model co-location via hierarchical weight caching and elastic memory provisioning. At the cluster level, it performs latency-aware routing to avoid cold-start delays and memory overloads. At the global level, it enforces workflow-aware prioritization to minimize head-of-line blocking for interactive tasks. Across prototype experiments and trace-driven simulations, Maestro reduces KV-reservation HBM by 67.2% and improves high-contention SLO attainment over EDF by 23.6 percentage points.

翻译：基于大语言模型的多智能体系统通过将复杂任务拆解为专业化智能体协作工作流，已成为处理复杂任务的重要范式。然而，大规模部署此类多智能体工作负载面临显著系统挑战。每个用户查询会触发由多次LLM调用构成的迭代流水线，相较于单轮查询大幅放大资源消耗。在资源受限的云计算环境中，这些工作流面临解码阶段非确定性且依赖输入的成本、存在内存碎片化与过度配置的重尾多模型需求，以及跨集群调度权衡问题。本文提出Maestro——专为严格GPU预算下LLM-MAS服务设计的工作负载感知调度系统。Maestro显式利用智能体语义与角色：预测每个阶段的输出长度与内存使用量，并以此驱动分层调度器。在节点层面，Maestro通过分层权重缓存与弹性内存配置实现动态多模型协同部署；在集群层面，其执行延迟感知路由以避免冷启动延迟与内存过载；在全局层面，实施工作流感知优先级排序以最小化交互式任务的队头阻塞。通过原型实验与流量驱动仿真，Maestro将KV预留HBM降低67.2%，并在高竞争场景下将SLO达标率较EDF提升23.6个百分点。