Large technology firms face the problem of moderating content on their online platforms for compliance with laws and policies. To accomplish this at the scale of billions of pieces of content per day, a combination of human and machine review are necessary to label content. Subjective judgement and human bias are of concern to both human annotated content as well as to auditors who may be employed to evaluate the quality of such annotations in conformance with law and/or policy. To address this concern, this paper presents a novel application of statistical analysis methods to identify human error and these sources of audit risk.

翻译：大型科技公司面临对其在线平台上的内容进行审核以符合法律及政策的挑战。为应对每日数十亿条内容的审核规模，需结合人工与机器审查来标注内容。主观判断与人为偏差问题不仅存在于人工标注内容中，也可能影响受雇评估此类标注质量（以确认是否符合法律法规或政策）的审计人员。针对该问题，本文提出一种新颖的统计分析方法应用，以识别人为错误及审计风险来源。