Dependabot, a popular dependency management tool, includes a compatibility score feature that helps client packages assess the risk of accepting a dependency update by leveraging knowledge from "the crowd". For each dependency update, Dependabot calculates this compatibility score as the proportion of successful updates performed by other client packages that use the same provider package as a dependency. In this paper, we study the efficacy of the compatibility score to help client packages assess the risks involved with accepting a dependency update. We analyze 579,206 pull requests opened by Dependabot to update a dependency, along with 618,045 compatibility score records calculated by Dependabot. We find that a compatibility score cannot be calculated for 83% of the dependency updates due to the lack of data from the crowd. Yet, the vast majority of the scores that can be calculated have a small confidence interval and are based on low-quality data, suggesting that client packages should have additional angles to evaluate the risk of an update and the trustworthiness of the compatibility score. To overcome these limitations, we propose metrics that amplify the input from the crowd and demonstrate the ability of those metrics to predict the acceptance of a successful update by client packages. We also demonstrate that historical update metrics from client packages can be used to provide a more personalized compatibility score. Based on our findings, we argue that, when leveraging the crowd, dependency management bots should include a confidence interval to help calibrate the trust clients can place in the compatibility score, and consider the quality of tests that exercise candidate updates.

翻译：Dependabot 是一款流行的依赖管理工具，其提供的兼容性评分功能通过利用“群体”知识，帮助客户端包评估接受依赖项更新的风险。对于每次依赖项更新，Dependabot 会计算此兼容性评分：它将同一提供者包作为依赖项的其他客户端包成功执行更新的比例。在本文中，我们研究了兼容性评分在帮助客户端包评估接受依赖项更新风险方面的有效性。我们分析了 Dependabot 为更新依赖项而提出的 579,206 个拉取请求，以及 Dependabot 计算的 618,045 条兼容性评分记录。我们发现，由于缺乏来自群体的数据，83% 的依赖项更新无法计算兼容性评分。然而，绝大多数可计算的评分置信区间较小，但基于低质量数据，这表明客户端包需要额外的视角来评估更新的风险以及兼容性评分的可信度。为克服这些局限，我们提出了能够放大群体输入的度量指标，并展示了这些指标预测客户端包接受成功更新的能力。此外，我们还证明，来自客户端包的历史更新度量指标可用于提供更具个性化的兼容性评分。基于我们的研究发现，我们认为，在利用群体智慧时，依赖管理机器人应包含置信区间，以帮助校准客户端对兼容性评分的信任程度，并考虑测试候选更新的质量。