Data generated by audits of social media websites have formed the basis of our understanding of the biases presented in algorithmic content recommendation systems. As legislators around the world are beginning to consider regulating the algorithmic systems that drive online platforms, it is critical to ensure the correctness of these inferred biases. However, as we will show in this paper, doing so is a challenging task for a variety of reasons related to the complexity of configuration parameters associated with the audits that gather data from a specific platform. Focusing specifically on YouTube, we show that conducting audits to make inferences about YouTube's recommendation systems is more methodologically challenging than one might expect. There are many methodological decisions that need to be considered in order to obtain scientifically valid results, and each of these decisions incur costs. For example, should an auditor use (expensive to obtain) logged-in YouTube accounts while gathering recommendations from the algorithm to obtain more accurate inferences? We explore the impact of this and many other decisions and make some startling discoveries about the methodological choices that impact YouTube's recommendations. Taken all together, our research suggests auditing configuration compromises that YouTube auditors and researchers can use to reduce audit overhead, both economically and computationally, without sacrificing accuracy of their inferences. Similarly, we also identify several configuration parameters that have a significant impact on the accuracy of measured inferences and should be carefully considered.

翻译：由社交媒体网站审计生成的数据构成了我们理解算法内容推荐系统中偏见的基础。随着全球立法者开始考虑监管驱动在线平台的算法系统，确保这些推断出的偏见的准确性至关重要。然而，正如本文将展示的，由于与从特定平台收集数据的审计相关的配置参数复杂性，这是一项具有挑战性的任务。我们专门针对YouTube，证明开展审计以推断YouTube推荐系统的行为在方法论上比预期更具挑战性。为了获得科学上有效的结果，需要考虑许多方法论决策，而每一项决策都会带来成本。例如，审计员在从算法收集推荐时是否应使用（获取成本高昂的）已登录YouTube账户以获得更准确的推断？我们探讨了这一决策及其他许多决策的影响，并对影响YouTube推荐的方法论选择做出了一些惊人发现。综合来看，我们的研究提出了一些审计配置方案，YouTube审计员和研究人员可利用这些方案在保持推断准确性的同时，从经济上和计算上降低审计开销。同样，我们还识别出几个对测量推断准确性有显著影响且应慎重考虑的配置参数。