Algorithm audits are powerful tools for studying black-box systems. While very effective in examining technical components, the method stops short of a sociotechnical frame, which would also consider users as an integral and dynamic part of the system. Addressing this gap, we propose the concept of sociotechnical auditing: auditing methods that evaluate algorithmic systems at the sociotechnical level, focusing on the interplay between algorithms and users as each impacts the other. Just as algorithm audits probe an algorithm with varied inputs and observe outputs, a sociotechnical audit (STA) additionally probes users, exposing them to different algorithmic behavior and measuring resulting attitudes and behaviors. To instantiate this method, we develop Intervenr, a platform for conducting browser-based, longitudinal sociotechnical audits with consenting, compensated participants. Intervenr investigates the algorithmic content users encounter online and coordinates systematic client-side interventions to understand how users change in response. As a case study, we deploy Intervenr in a two-week sociotechnical audit of online advertising (N=244) to investigate the central premise that personalized ad targeting is more effective on users. In the first week, we collect all browser ads delivered to users, and in the second, we deploy an ablation-style intervention that disrupts normal targeting by randomly pairing participants and swapping all their ads. We collect user-oriented metrics (self-reported ad interest and feeling of representation) and advertiser-oriented metrics (ad views, clicks, and recognition) throughout, along with a total of over 500,000 ads. Our STA finds that targeted ads indeed perform better with users, but also that users begin to acclimate to different ads in only a week, casting doubt on the primacy of personalized ad targeting given the impact of repeated exposure.

翻译：算法审计是研究黑箱系统的有力工具。尽管该方法在检验技术组件方面成效显著，但其未能触及社会技术框架——该框架将用户视为系统不可或缺的动态组成部分。为弥补这一不足，我们提出"社会技术审计"概念：一种在社会技术层面评估算法系统的审计方法，聚焦算法与用户之间的相互影响。正如算法审计通过多样化输入探测算法并观察输出，社会技术审计（STA）额外对用户进行探测，使其暴露于不同算法行为下，并测量由此产生的态度与行为变化。为实例化该方法，我们开发了Intervenr平台，用于在浏览器端对知情同意并获补偿的参与者开展纵向社会技术审计。该平台通过追踪用户在线遇到的算法内容，协调系统端的客户端干预措施，以理解用户如何因应变化而改变。作为案例研究，我们利用Intervenr对网络广告（N=244）开展为期两周的社会技术审计，检验"个性化广告定向对用户更有效"的核心假设。第一周，我们收集用户浏览器接收的所有广告；第二周，部署消融式干预——随机配对参与者并互换其所有广告以破坏正常定向。全程收集面向用户的指标（自报广告兴趣与表征感受）及面向广告主的指标（广告浏览数、点击数与识别率），累计获取超过50万条广告。STA发现，定向广告确实对用户表现更佳，但用户仅在一周内便开始适应不同广告，这使人们对个性化广告定向在重复曝光影响下的主导地位产生质疑。