Large enterprises often operate extensive Continuous Integration (CI) pipelines on large, heterogeneous compute clusters, where conservative, statically defined resource requirements are used to ensure build reliability. This practice leads to substantial system memory over-allocation, reduced cluster utilization, and increased operational costs. In this paper, we motivate the need for intelligent resource prediction by analyzing over 300,000 historical build executions from a production CI environment with more than one thousand compute nodes. Our analysis shows that, on average, more than 60% of allocated system memory remains unused. We then compare multiple machine learning approaches for predicting build task memory usage, including classification-based methods and regression-based quantile prediction. Our final solution employs a LightGBM-XGBoost quantile regression ensemble optimized to minimize under-allocation while reducing over-provisioning. We integrate this solution into the production CI pipeline via a microservice-based orchestration layer, achieving average memory savings of approximately 36GB per build and reducing under-allocation rates to below 0.3% without negatively impacting build execution times.

翻译：大型企业通常会在大规模异构计算集群上运行复杂的持续集成（CI）流水线，其中采用保守的静态定义资源需求来确保构建可靠性。这种做法会导致系统内存严重过度分配、集群利用率降低以及运营成本增加。本文通过分析拥有超过一千个计算节点的生产CI环境中30余万次历史构建执行，论证了智能资源预测的必要性。分析表明，平均超过60%的已分配系统内存未被使用。随后，我们比较了多种用于预测构建任务内存使用量的机器学习方法，包括基于分类的方法和基于回归的分位数预测。最终方案采用了针对LightGBM-XGBoost分位数回归集成优化，旨在最小化欠分配同时减少过度配置。我们通过基于微服务的编排层将该方案集成到生产CI流水线中，实现了每次构建平均约36GB的内存节省，并将欠分配率降至0.3%以下，且未对构建执行时间产生负面影响。