Modern enterprise AI applications increasingly rely on compound AI systems - architectures that compose multiple models, retrievers, and tools to accomplish complex tasks. Deploying such systems in production demands inference infrastructure that can efficiently serve concurrent, heterogeneous model invocations while maintaining cost-effectiveness and low latency. This paper presents a production deployment study of a modular, platform-agnostic inference architecture developed at Salesforce to support compound AI use cases including Agentforce (autonomous AI agents) and ApexGuru (AI-powered code analysis). The system integrates serverless execution, dynamic autoscaling, and MLOps pipelines to deliver consistent low-latency inference across multi-component agent workflows. We report production results demonstrating over 50% reduction in tail latency (P95), up to 3.9x throughput improvement, and 30 to 40% cost savings compared to prior static deployments. We further present a novel analysis of compound-system-specific challenges including multi-model fan-out overhead, cascading cold-start propagation, and heterogeneous scaling dynamics that emerge uniquely when serving agentic workloads. Through detailed case studies and operational lessons, we illustrate how the architecture enables compound AI systems to scale model invocations in parallel, handle bursty multi-agent workloads, and support rapid model iteration - capabilities essential for operationalizing agentic AI at enterprise scale.

翻译：现代企业AI应用日益依赖复合AI系统——即组合多个模型、检索器和工具以完成复杂任务的架构。在生产环境中部署此类系统需要推理基础设施能够高效服务并发、异构的模型调用，同时保持成本效益和低延迟。本文介绍了Salesforce开发的一种模块化、平台无关的推理架构的生产部署研究，该架构支持包含Agentforce（自主AI智能体）和ApexGuru（AI驱动代码分析）在内的复合AI用例。系统集成了无服务器执行、动态自动扩缩容和MLOps流水线，在多组件智能体工作流中实现了持续的低延迟推理。我们报告的生产结果表明，与先前的静态部署相比，尾延迟（P95）降低超过50%，吞吐量提升高达3.9倍，成本节约30%至40%。我们进一步提出对复合系统特有挑战的新颖分析，包括多模型扇出开销、级联冷启动传播，以及在服务智能体工作负载时特有的异构扩缩容动态。通过详细的案例研究和运营经验，我们阐释了该架构如何使复合AI系统能够并行扩展模型调用、处理突发的多智能体工作负载，并支持快速的模型迭代——这些能力对于在企业规模下实现智能体AI的运营化至关重要。