This paper presents a solution to the challenge of mitigating carbon emissions from hosting large-scale machine learning (ML) inference services. ML inference is critical to modern technology products, but it is also a significant contributor to carbon footprint. We introduce Clover, a carbon-friendly ML inference service runtime system that balances performance, accuracy, and carbon emissions through mixed-quality models and GPU resource partitioning. Our experimental results demonstrate that Clover is effective in substantially reducing carbon emissions while maintaining high accuracy and meeting service level agreement (SLA) targets.

翻译：本文提出了一种解决方案，用于缓解大规模机器学习推理服务托管过程中碳排放带来的挑战。机器学习推理是现代技术产品中的关键环节，但同时也是碳足迹的重要来源。我们介绍了Clover——一种碳友好的机器学习推理服务运行时系统，该系统通过混合质量模型与GPU资源分区，在性能、准确性和碳排放之间取得平衡。实验结果表明，Clover在显著减少碳排放的同时，能够保持高准确性并满足服务等级协议目标。