Algorithmic stability is a concept from learning theory that expresses the degree to which changes to the input data (e.g., removal of a single data point) may affect the outputs of a regression algorithm. Knowing an algorithm's stability properties is often useful for many downstream applications -- for example, stability is known to lead to desirable generalization properties and predictive inference guarantees. However, many modern algorithms currently used in practice are too complex for a theoretical analysis of their stability properties, and thus we can only attempt to establish these properties through an empirical exploration of the algorithm's behavior on various data sets. In this work, we lay out a formal statistical framework for this kind of "black box testing" without any assumptions on the algorithm or the data distribution, and establish fundamental bounds on the ability of any black box test to identify algorithmic stability.

翻译：算法稳定性是一个概念,它来自学习理论,表明输入数据的变化(例如删除一个数据点)在多大程度上会影响回归算法的输出。了解算法的稳定性属性对于许多下游应用来说往往有用,例如,据知稳定性会导致可取的概括性属性和预测推论保证。然而,目前实践中使用的许多现代算法对于其稳定性属性的理论分析来说过于复杂,因此,我们只能尝试通过对各种数据集的算法行为进行实证性探索来确定这些属性。在这项工作中,我们为这种“黑盒测试”制定了正式的统计框架,而没有关于算法或数据分布的任何假设,并确定了任何黑盒测试确定算法稳定性的能力的基本界限。