Responsible design of AI systems is a shared goal across HCI and AI communities. Responsible AI (RAI) tools have been developed to support practitioners to identify, assess, and mitigate ethical issues during AI development. These tools take many forms (e.g., design playbooks, software toolkits, documentation protocols). However, research suggests that use of RAI tools is shaped by organizational contexts, raising questions about how effective such tools are in practice. To better understand how RAI tools are -- and might be -- evaluated, we conducted a qualitative analysis of 37 publications that discuss evaluations of RAI tools. We find that most evaluations focus on usability, while questions of tools' effectiveness in changing AI development are sidelined. While usability evaluations are an important approach to evaluate RAI tools, we draw on evaluation approaches from other fields to highlight developer- and community-level steps to support evaluations of RAI tools' effectiveness in shaping AI development practices and outcomes.

翻译：负责任的AI系统设计是人机交互与人工智能领域的共同目标。为支持从业者在AI开发过程中识别、评估和缓解伦理问题，现已开发出负责任AI工具。这些工具形式多样（如设计手册、软件工具包、文档协议）。然而研究表明，负责任AI工具的使用受组织情境影响，这引发了关于此类工具在实践中有效性的疑问。为深入理解负责任AI工具当前及潜在的评估方式，我们对37篇讨论负责任AI工具评估的出版物进行了定性分析。结果发现，大多数评估聚焦于可用性，而工具在改变AI开发过程中的有效性问题却被边缘化。虽然可用性评估是评估负责任AI工具的重要方法，但我们借鉴其他领域的评估方法，从开发者层面和社区层面提出了支持评估负责任AI工具在塑造AI开发实践和成果方面的有效性的步骤。