Man-at-the-end (MATE) attackers have full control over the system on which the attacked software runs, and try to break the confidentiality or integrity of assets embedded in the software. Both companies and malware authors want to prevent such attacks. This has driven an arms race between attackers and defenders, resulting in a plethora of different protection and analysis methods. However, it remains difficult to measure the strength of protections because MATE attackers can reach their goals in many different ways and a universally accepted evaluation methodology does not exist. This survey systematically reviews the evaluation methodologies of papers on obfuscation, a major class of protections against MATE attacks. For 572 papers, we collected 113 aspects of their evaluation methodologies, ranging from sample set types and sizes, over sample treatment, to performed measurements. We provide detailed insights into how the academic state of the art evaluates both the protections and analyses thereon. In summary, there is a clear need for better evaluation methodologies. We identify nine challenges for software protection evaluations, which represent threats to the validity, reproducibility, and interpretation of research results in the context of MATE attacks.

翻译：终端攻击者（MATE）可完全控制被攻击软件所运行的系统，并试图破坏软件中嵌入资产的机密性或完整性。无论是企业还是恶意软件作者都希望阻止此类攻击。这推动了攻击者与防御者之间的军备竞赛，催生了大量不同的保护与分析技术。然而，由于MATE攻击者可通过多种途径达成目标，且缺乏普遍接受的评估方法论，衡量保护强度的难度依然存在。本综述系统性地梳理了针对混淆技术（抵御MATE攻击的主要防护类别）的研究论文中的评估方法论。针对572篇论文，我们从样本集类型与规模、样本处理方式到执行测量等维度，收集了其评估方法论中的113项特征。我们深入揭示了学术界在评估保护技术及其分析方法时的现状。总体而言，当前亟需更完善的评估方法论。我们识别出软件保护评估面临的九大挑战，这些挑战对MATE攻击背景下研究结果的效度、可重复性及解读构成了威胁。