For online health communities, community trust is paramount. Yet, advances in Large Language Models (LLMs) generating advice may erode this trust, especially if users cannot identify whether LLMs have been used. We investigate the feasibility of community-based detection of health advice authorship and how self-moderation of LLMs could help enhance advice utilization. In an online experiment, we evaluate people's ability to distinguish AI-generated from human-written advice across two health conditions, considering lived experience with a condition, AI-recognition training, and user attitudes towards transparency and trust around AI use. Our results indicate the need for transparency coupled with trust. We find little evidence of people's ability to discern advice authorship. However, we find a consistent effect of the health condition. Our qualitative findings identify unreliable signals, resulting in flawed heuristic evaluations of the advice. Our findings point to opportunities to improve the self-moderation of LLM-based AI and aid community-based AI moderation.

翻译：对于在线健康社区而言，社区信任至关重要。然而，大语言模型生成建议的进步可能会削弱这种信任，尤其是当用户无法识别是否使用了LLM时。我们研究了基于社区的健康建议作者身份检测的可行性，以及LLM的自我审核如何帮助提升建议的利用率。在一项在线实验中，我们评估了人们在两种健康状况下区分AI生成建议与人类撰写建议的能力，考虑了个人对健康状况的亲身体验、AI识别训练以及用户对AI使用透明度与信任的态度。我们的结果表明，透明度与信任的结合是必要的。我们几乎没有发现人们具备辨别建议作者身份能力的证据。然而，我们发现健康状况具有一致的影响。我们的定性研究识别出不可靠的信号，导致对建议的启发式评估存在缺陷。我们的研究结果为改进基于LLM的AI的自我审核以及辅助社区层面的AI审核提供了机会。