In some problem spaces, the high cost of obtaining ground truth labels necessitates use of lower quality reference datasets. It is difficult to benchmark model performance using these datasets, as evaluation results may be biased. We propose a supplement to using reference labels, which we call an approximate ground truth refinement (AGTR). Using an AGTR, we prove that bounds on specific metrics used to evaluate clustering algorithms and multi-class classifiers can be computed without reference labels. We also introduce a procedure that uses an AGTR to identify inaccurate evaluation results produced from datasets of dubious quality. Creating an AGTR requires domain knowledge, and malware family classification is a task with robust domain knowledge approaches that support the construction of an AGTR. We demonstrate our AGTR evaluation framework by applying it to a popular malware labeling tool to diagnose over-fitting in prior testing and evaluate changes whose impact could not be meaningfully quantified under previous data.

翻译：在某些问题领域,获取地面真相标签的成本高昂,需要使用质量较低的参考数据集,很难用这些数据集来衡量模型性能,因为评价结果可能有偏差。我们建议对使用参考标签进行补充,我们称之为近似地面真相改进(AGTR)。我们使用AGTR,证明可以不使用参考标签来计算用于评价集群算法和多级分类器的具体指标的界限。我们还采用一个程序,利用AGTR来查明从质量可疑的数据集中得出的不准确的评价结果。建立AGTR需要域知识,而恶意软件家庭分类是一项具有可靠领域知识的任务,支持AGTR的构建。我们通过将AGTR评估框架应用到普通的恶意软件标签工具来诊断在先前测试和评价那些影响无法在先前数据下有意义地量化的变化时是否过度适用。