Identifying patients who benefit from a treatment is a key aspect of personalized medicine, which allows the development of individualized treatment rules (ITRs). Many machine learning methods have been proposed to create such rules. However, to what extent the methods lead to similar ITRs, i.e., recommending the same treatment for the same individuals is unclear. In this work, we compared 22 of the most common approaches in two randomized control trials. Two classes of methods can be distinguished. The first class of methods relies on predicting individualized treatment effects from which an ITR is derived by recommending the treatment evaluated to the individuals with a predicted benefit. In the second class, methods directly estimate the ITR without estimating individualized treatment effects. For each trial, the performance of ITRs was assessed by various metrics, and the pairwise agreement between all ITRs was also calculated. Results showed that the ITRs obtained via the different methods generally had considerable disagreements regarding the patients to be treated. A better concordance was found among akin methods. Overall, when evaluating the performance of ITRs in a validation sample, all methods produced ITRs with limited performance, suggesting a high potential for optimism. For non-parametric methods, this optimism was likely due to overfitting. The different methods do not lead to similar ITRs and are therefore not interchangeable. The choice of the method strongly influences for which patients a certain treatment is recommended, drawing some concerns about their practical use.

翻译：识别能从治疗中获益的患者是个体化医学的关键方面，这促进了个体化治疗规则（ITRs）的发展。许多机器学习方法已被提出用于创建此类规则。然而，这些方法在多大程度上产生相似的ITRs（即对相同个体推荐相同治疗）尚不明确。在本研究中，我们通过两项随机对照试验比较了22种最常用的方法。可区分两类方法：第一类方法依赖于预测个体化治疗效果，从中推导出ITR，即对预测获益的个体推荐评估的治疗；第二类方法直接估计ITR，而不估计个体化治疗效果。每项试验中，通过多种指标评估ITRs的性能，并计算所有ITRs之间的成对一致性。结果显示，不同方法获得的ITRs在待治疗患者方面通常存在显著分歧。同类方法间的一致性较高。总体而言，在验证样本中评估ITRs性能时，所有方法产生的ITRs性能有限，表明存在高度乐观偏差。对于非参数方法，这种乐观偏差可能源于过拟合。不同方法不会产生相似的ITRs，因此不可互换。方法的选择强烈影响对哪些患者推荐特定治疗，这对其实际应用提出了一些担忧。