Resilience against stragglers is a critical element of prediction serving systems, tasked with executing inferences on input data for a pre-trained machine-learning model. In this paper, we propose NeRCC, as a general straggler-resistant framework for approximate coded computing. NeRCC includes three layers: (1) encoding regression and sampling, which generates coded data points, as a combination of original data points, (2) computing, in which a cluster of workers run inference on the coded data points, (3) decoding regression and sampling, which approximately recovers the predictions of the original data points from the available predictions on the coded data points. We argue that the overall objective of the framework reveals an underlying interconnection between two regression models in the encoding and decoding layers. We propose a solution to the nested regressions problem by summarizing their dependence on two regularization terms that are jointly optimized. Our extensive experiments on different datasets and various machine learning models, including LeNet5, RepVGG, and Vision Transformer (ViT), demonstrate that NeRCC accurately approximates the original predictions in a wide range of stragglers, outperforming the state-of-the-art by up to 23%.

翻译：摘要：抵御掉队者（stragglers）是预测服务系统中的关键要素，该系统负责对预训练机器学习模型的输入数据执行推理。本文提出NeRCC，一种通用的抗掉队者近似编码计算框架。NeRCC包含三层：(1) 编码回归与采样，通过组合原始数据点生成编码数据点；(2) 计算，由一组工作节点对编码数据点执行推理；(3) 解码回归与采样，基于编码数据点上已有的预测结果近似恢复原始数据点的预测值。我们论证该框架的总体目标揭示了编码层与解码层中两个回归模型之间的内在联系。通过归纳两个正则化项对嵌套回归问题的依赖性，并提出联合优化方案，我们解决了嵌套回归问题。在不同数据集及多种机器学习模型（包括LeNet5、RepVGG和Vision Transformer (ViT)）上的大量实验表明，NeRCC能在广泛的掉队者场景下精确逼近原始预测结果，性能较现有最优方法提升高达23%。