In education, health, and human services, an intervention program is usually implemented by many local organizations. Determining which organizations are more effective is essential for theoretically characterizing effective practices and for intervening to enhance the capacity of ineffective organizations. In multisite randomized trials, site-specific intention-to-treat (ITT) effects are likely invalid indicators for organizational effectiveness and may lead to inequitable decisions. This is because sites differ in their local ecological conditions including client composition, alternative programs, and community context. Applying the potential outcomes framework, this study proposes a mathematical definition for the relative effectiveness of an organization. The estimand contrasts the performance of a focal organization with those that share the features of its local ecological conditions. The identification relies on relatively weak assumptions by leveraging observed control group outcomes that capture the confounding impacts of alternative programs and community context. We propose a two-step mixed-effects modeling (2SME) procedure. Simulations demonstrate significant improvements when compared with site-specific ITT analyses or analyses that only adjust for between-site differences in the observed baseline participant composition. We illustrate its use through an evaluation of the relative effectiveness of individual Job Corps centers by reanalyzing data from the National Job Corps Study, a multisite randomized trial that included 100 Job Corps centers nationwide serving disadvantaged youths. The new strategy promises to alleviate consequential misclassifications of some of the most effective Job Corps centers as least effective and vice versa.

翻译：在教育、健康及人类服务领域，干预项目通常由众多地方组织共同实施。确定哪些组织更具效能，对于从理论上描述有效实践以及通过干预提升低效组织能力至关重要。在多站点随机试验中，站点特定的意向治疗效应往往不能作为组织效能的有效指标，并可能导致不公平的决策。这是因为各站点在本地生态条件（包括服务对象构成、替代项目及社区环境）上存在差异。本研究基于潜在结果框架，提出了组织相对效能的数学定义。该估计量通过对比目标组织与具有相似本地生态条件特征的其他组织的表现来实现。识别过程依赖于相对较弱的假设，通过利用观测到的对照组结果来捕捉替代项目和社区环境的混杂影响。我们提出了一种两步混合效应建模程序。模拟实验表明，相较于站点特异性意向治疗效应分析或仅针对观测基线参与者构成进行站点间差异调整的分析方法，本方法具有显著改进。我们通过重新分析全国职业培训团研究的数据，评估了各职业培训中心的相对效能，以此说明该方法的应用。该全国性多站点随机试验覆盖了100个为弱势青年提供服务的职业培训中心。新策略有望缓解将部分最有效的职业培训中心误判为最低效中心（反之亦然）所导致的重大分类错误。