Large-scale prediction models (typically using tools from artificial intelligence, AI, or machine learning, ML) are increasingly ubiquitous across a variety of industries and scientific domains. Such methods are often paired with detailed data from sources such as electronic health records, wearable sensors, and omics data (high-throughput technology used to understand biology). Despite their utility, implementing AI and ML tools at the scale necessary to work with this data introduces two major challenges. First, it can cost tens of thousands of dollars to train a modern AI/ML model at scale. Second, once the model is trained, its predictions may become less relevant as patient and provider behavior change, and predictions made for one geographical area may be less accurate for another. These two challenges raise a fundamental question: how often should you refit the AI/ML model to optimally trade-off between cost and relevance? Our work provides a framework for making decisions about when to {\it refit} AI/ML models when the goal is to maintain valid statistical inference (e.g. estimating a treatment effect in a clinical trial). Drawing on portfolio optimization theory, we treat the decision of {\it recalibrating} versus {\it refitting} the model as a choice between ''investing'' in one of two ''assets.'' One asset, recalibrating the model based on another model, is quick and relatively inexpensive but bears uncertainty from sampling and the possibility that the other model is not relevant to current circumstances. The other asset, {\it refitting} the model, is costly but removes the irrelevance concern (though not the risk of sampling error). We explore the balancing act between these two potential investments in this paper.

翻译：大规模预测模型（通常采用人工智能或机器学习工具）在各类工业与科学领域日益普及。此类方法常与电子健康记录、可穿戴传感器及组学数据等高通量生物技术提供的详细数据结合使用。尽管具有实用性，但为处理此类数据而实施相应规模的人工智能与机器学习工具仍面临两大挑战：其一，大规模训练现代人工智能/机器学习模型的成本可达数万美元；其二，模型训练完成后，其预测结果可能因患者和医疗服务提供者行为变化而逐渐失效，且针对特定地理区域的预测在其他区域可能准确性下降。这两大挑战引出一个根本性问题：应如何确定人工智能/机器学习模型的重拟合频率，以实现成本与时效性的最优权衡？本研究构建了一个决策框架，用于在需要保持有效统计推断（例如临床试验中估计治疗效果）的目标下，确定何时对人工智能/机器学习模型进行重拟合。借鉴投资组合优化理论，我们将模型重新校准与重拟合的决策视为两种"资产"间的投资选择：一种资产是基于其他模型进行快速且成本相对较低的重新校准，但需承担抽样不确定性及其他模型可能不适应当前情境的风险；另一种资产是进行成本高昂但能消除时效性问题的重拟合（尽管无法规避抽样误差风险）。本文深入探讨了这两种潜在投资选择间的平衡机制。