Identifying potential social and ethical risks in emerging machine learning (ML) models and their applications remains challenging. In this work, we applied two well-established safety engineering frameworks (FMEA, STPA) to a case study involving text-to-image models at three stages of the ML product development pipeline: data processing, integration of a T2I model with other models, and use. Results of our analysis demonstrate the safety frameworks - both of which are not designed explicitly examine social and ethical risks - can uncover failure and hazards that pose social and ethical risks. We discovered a broad range of failures and hazards (i.e., functional, social, and ethical) by analyzing interactions (i.e., between different ML models in the product, between the ML product and user, and between development teams) and processes (i.e., preparation of training data or workflows for using an ML service/product). Our findings underscore the value and importance of examining beyond an ML model in examining social and ethical risks, especially when we have minimal information about an ML model.

翻译：识别新兴机器学习模型及其应用中潜在的社会和伦理风险仍然具有挑战性。在本研究中，我们将两个成熟的安全工程框架（FMEA、STPA）应用于一个涉及文本到图像模型的案例研究，涵盖机器学习产品开发管道的三个阶段：数据处理、文本到图像模型与其他模型的集成以及使用。我们的分析结果表明，这些安全框架——尽管并非专门设计用于检查社会和伦理风险——能够揭示构成社会和伦理风险的故障和危害。通过分析交互（即产品中不同机器学习模型之间、机器学习产品与用户之间以及开发团队之间）和流程（即训练数据准备或使用机器学习服务/产品的工作流程），我们发现了一系列广泛的故障和危害（即功能、社会和伦理方面的）。我们的发现强调了在检查社会和伦理风险时，超越机器学习模型本身进行审视的价值和重要性，尤其是在我们对机器学习模型了解有限的情况下。