As conversational AI systems increasingly permeate the socio-emotional realms of human life, they bring both benefits and risks to individuals and society. Despite extensive research on detecting and categorizing harms in AI systems, less is known about the harms that arise from social interactions with AI chatbots. Through a mixed-methods analysis of 35,390 conversation excerpts shared on r/replika, an online community for users of the AI companion Replika, we identified six categories of harmful behaviors exhibited by the chatbot: relational transgression, verbal abuse and hate, self-inflicted harm, harassment and violence, mis/disinformation, and privacy violations. The AI contributes to these harms through four distinct roles: perpetrator, instigator, facilitator, and enabler. Our findings highlight the relational harms of AI chatbots and the danger of algorithmic compliance, enhancing the understanding of AI harms in socio-emotional interactions. We also provide suggestions for designing ethical and responsible AI systems that prioritize user safety and well-being.

翻译：随着对话式人工智能系统日益渗透到人类生活的社会情感领域，它们为个人和社会既带来益处也带来风险。尽管已有大量关于人工智能系统中危害检测与分类的研究，但对于与AI聊天机器人社交互动所产生的危害仍知之甚少。通过对AI伴侣Replika用户在线社区r/replika上分享的35,390条对话摘录进行混合方法分析，我们识别出该聊天机器人表现出的六类有害行为：关系越界、言语辱骂与仇恨、自我伤害、骚扰与暴力、错误/虚假信息以及隐私侵犯。AI通过四种不同角色促成这些危害：施害者、煽动者、促进者和纵容者。我们的研究结果凸显了AI聊天机器人带来的关系性危害以及算法顺从性的危险，深化了对社会情感互动中AI危害的理解。我们还为设计优先考虑用户安全与福祉的伦理且负责任的人工智能系统提供了建议。