Extrapolation is crucial in many statistical and machine learning applications, as it is common to encounter test data outside the training support. However, extrapolation is a considerable challenge for nonlinear models. Conventional models typically struggle in this regard: while tree ensembles provide a constant prediction beyond the support, neural network predictions tend to become uncontrollable. This work aims at providing a nonlinear regression methodology whose reliability does not break down immediately at the boundary of the training support. Our primary contribution is a new method called `engression' which, at its core, is a distributional regression technique for pre-additive noise models, where the noise is added to the covariates before applying a nonlinear transformation. Our experimental results indicate that this model is typically suitable for many real data sets. We show that engression can successfully perform extrapolation under some assumptions such as a strictly monotone function class, whereas traditional regression approaches such as least-squares regression and quantile regression fall short under the same assumptions. We establish the advantages of engression over existing approaches in terms of extrapolation, showing that engression consistently provides a meaningful improvement. Our empirical results, from both simulated and real data, validate these findings, highlighting the effectiveness of the engression method. The software implementations of engression are available in both R and Python.

翻译：摘要：外推在许多统计和机器学习应用中至关重要，因为测试数据超出训练支持范围的情况十分常见。然而，外推对于非线性模型来说是一项重大挑战。传统方法通常在此方面表现不佳：树集成模型在支持范围外给出恒定预测，而神经网络预测则往往变得难以控制。本文旨在提出一种非线性回归方法，其在训练支持边界处不会立即失效。我们的主要贡献是一种名为"engression"的新方法，其核心是针对预加性噪声模型（噪声先添加到协变量中再进行非线性变换）的分布回归技术。实验结果表明，该模型通常适用于许多真实数据集。我们证明，在严格单调函数类等假设下，engression能成功实现外推，而传统回归方法（如最小二乘回归和分位数回归）在相同假设下则表现不足。我们确立了engression在可外推性方面相对于现有方法的优势，表明engression能持续提供有意义的改进。基于模拟数据和真实数据的实验结果均验证了这些发现，凸显了engression方法的有效性。engression的软件实现已提供R和Python版本。