Extrapolation is crucial in many statistical and machine learning applications, as it is common to encounter test data outside the training support. However, extrapolation is a considerable challenge for nonlinear models. Conventional models typically struggle in this regard: while tree ensembles provide a constant prediction beyond the support, neural network predictions tend to become uncontrollable. This work aims at providing a nonlinear regression methodology whose reliability does not break down immediately at the boundary of the training support. Our primary contribution is a new method called `engression' which, at its core, is a distributional regression technique for pre-additive noise models, where the noise is added to the covariates before applying a nonlinear transformation. Our experimental results indicate that this model is typically suitable for many real data sets. We show that engression can successfully perform extrapolation under some assumptions such as a strictly monotone function class, whereas traditional regression approaches such as least-squares regression and quantile regression fall short under the same assumptions. We establish the advantages of engression over existing approaches in terms of extrapolation, showing that engression consistently provides a meaningful improvement. Our empirical results, from both simulated and real data, validate these findings, highlighting the effectiveness of the engression method. The software implementations of engression are available in both R and Python.

翻译：摘要：外推在许多统计和机器学习应用中至关重要，因为测试数据常常超出训练数据的支持范围。然而，对于非线性模型而言，外推是一项重大挑战。传统模型在此方面通常表现不佳：树集成模型在支持范围外提供恒定预测，而神经网络预测则往往变得不可控。本文旨在提出一种非线性回归方法，其可靠性不会在训练支持边界处立即失效。我们的主要贡献是一种名为"Engression"的新方法，其核心是一种针对预加性噪声模型（即在应用非线性变换前向协变量添加噪声）的分布回归技术。实验结果表明，该模型通常适用于许多真实数据集。我们证明，在严格单调函数类等假设下，Engression能够成功执行外推，而传统回归方法（如最小二乘回归和分位数回归）在相同假设下则表现不足。我们展示了Engression在外推方面相较于现有方法的优势，表明Engression能持续提供显著改进。基于模拟数据和真实数据的实证结果验证了这些发现，凸显了Engression方法的有效性。Engression方法的软件实现已提供R语言和Python版本。