Our goal is to improve reliability of Machine Learning (ML) systems deployed in the wild. ML models perform exceedingly well when test examples are similar to train examples. However, real-world applications are required to perform on any distribution of test examples. Current ML systems can fail silently on test examples with distribution shifts. In order to improve reliability of ML models due to covariate or domain shift, we propose algorithms that enable models to: (a) generalize to a larger family of test distributions, (b) evaluate accuracy under distribution shifts, (c) adapt to a target distribution. We study causes of impaired robustness to domain shifts and present algorithms for training domain robust models. A key source of model brittleness is due to domain overfitting, which our new training algorithms suppress and instead encourage domain-general hypotheses. While we improve robustness over standard training methods for certain problem settings, performance of ML systems can still vary drastically with domain shifts. It is crucial for developers and stakeholders to understand model vulnerabilities and operational ranges of input, which could be assessed on the fly during the deployment, albeit at a great cost. Instead, we advocate for proactively estimating accuracy surfaces over any combination of prespecified and interpretable domain shifts for performance forecasting. We present a label-efficient estimation to address estimation over a combinatorial space of domain shifts. Further, when a model's performance on a target domain is found to be poor, traditional approaches adapt the model using the target domain's resources. Standard adaptation methods assume access to sufficient labeled resources, which may be impractical for deployed models. We initiate a study of lightweight adaptation techniques with only unlabeled data resources with a focus on language applications.

翻译：我们的目标是提高部署在野外环境中的机器学习系统的可靠性。当测试样本与训练样本相似时，机器学习模型表现极其出色。然而，实际应用要求模型能够在任意分布的测试样本上表现良好。当前的机器学习系统在面对分布偏移的测试样本时，可能会出现静默失败。为了改善机器学习模型应对协变量或领域偏移的可靠性，我们提出了使模型具备以下能力的算法：(a) 泛化到更广泛的测试分布族，(b) 评估在分布偏移下的准确率，(c) 适应目标分布。我们研究了导致领域偏移鲁棒性受损的原因，并提出了训练领域鲁棒模型的算法。模型脆弱性的一个关键来源是领域过拟合，我们的新训练算法抑制了这种现象，转而鼓励领域通用的假设。尽管我们在某些问题设置下比标准训练方法提升了鲁棒性，但机器学习系统的性能仍然可能因领域偏移而剧烈变化。因此，开发者和利益相关者理解模型漏洞和输入的运行范围至关重要，这可以在部署过程中即时评估，尽管代价高昂。我们主张主动估计在预定义且可解释的领域偏移任意组合下的准确率曲面，以实现性能预测。我们提出了一种标签高效的估计方法，以解决领域偏移组合空间上的估计问题。此外，当模型在目标域上的性能被判定为较差时，传统方法会利用目标域的资源来调整模型。标准的适应方法假设可以访问充足的标注资源，这对于已部署的模型可能不切实际。我们率先研究了仅使用无标注数据资源的轻量级适应技术，重点关注语言应用场景。