Flaky tests are problematic because they non-deterministically pass or fail for the same software version under test, causing confusion and wasting development effort. While machine learning models have been used to predict flakiness and its root causes, there is much less work on providing support to fix the problem. To address this gap, in this paper, we focus on predicting the type of fix that is required to remove flakiness and then repair the test code on that basis. We do this for a subset of flaky test cases where the root cause of flakiness is in the test case itself and not in the production code. Our key idea is to guide the repair process with additional knowledge about the test's flakiness in the form of its predicted fix category. Thus, we first propose a framework that automatically generates labeled datasets for 13 fix categories and trains models to predict the fix category of a flaky test by analyzing the test code only. Our experimental results using code models and few-shot learning show that we can correctly predict most of the fix categories. To show the usefulness of such fix category labels for automatically repairing flakiness, in addition to informing testers, we augment a Large Language Model (LLM) like GPT with such extra knowledge to ask the LLM for repair suggestions. The results show that our suggested fix category labels, complemented with in-context learning, significantly enhance the capability of GPT 3.5 Turbo in generating fixes for flaky tests. Based on the execution and analysis of a sample of GPT-repaired flaky tests, we estimate that a large percentage of such repairs, (roughly between 70% and 90%) can be expected to pass. For the failing repaired tests, on average, 16% of the test code needs to be further changed for them to pass.

翻译：脆弱性测试（Flaky tests）因在同一软件版本下非确定性通过或失败，导致混淆并浪费开发资源，从而引发问题。尽管已有机器学习模型用于预测脆弱性及其根本原因，但在提供修复问题支持方面的研究仍显不足。为填补这一空白，本文聚焦于预测消除脆弱性所需的修复类型，并在此基础上修复测试代码。我们针对根本原因存在于测试用例本身而非生产代码的一类脆弱性测试用例展开研究。核心思路在于：通过预测修复类别为测试的脆弱性提供额外知识，从而引导修复过程。为此，我们首先提出一个框架，可自动生成包含13种修复类别的标注数据集，并训练模型仅通过分析测试代码预测脆弱性测试的修复类别。基于代码模型与少样本学习的实验结果表明，该方法能正确预测大多数修复类别。为验证修复类别标签对自动修复脆弱性的实用性（除告知测试人员外），我们向大语言模型（如GPT）注入此类额外知识，以获取修复建议。结果显示，所提出的修复类别标签结合情境学习，显著提升了GPT 3.5 Turbo在生成脆弱性测试修复方案上的能力。通过对GPT修复的脆弱性测试样本进行执行与分析，我们估计其中大部分修复（约70%至90%）预计可通过测试。对于修复后仍失败的测试，平均需进一步修改16%的测试代码方能通过。