Social media text data are often used to train Machine Learning (ML) models to identify users exhibiting high-risk mental health behaviors. However, sharing this sensitive data poses privacy risks and limits the growth of benchmark datasets. We comprehensively evaluate whether privacy-preserving ML techniques can enable safer data sharing while preserving performance. Specifically, we apply federated learning (FL) and Differentially Private FL for two widely-studied mental health prediction tasks: depression detection on X (Twitter) and suicide crisis detection on Reddit. We simulate realistic data-sharing scenarios by treating each user as a client in a non-IID setting, evaluating across different client fractions, aggregation strategies, and privacy budgets. While FL achieves comparable performance to centralized training (centralized F1 = 85.63; best FL model F1 = 83.16) on depression identification, we find that Differentially Private FL has a large performance-privacy trade-off (up to F1 = 27.01 drop) even with low levels of noise (epsilon = 50). This is due to the distortion of highly informative yet sparse mental health linguistic markers related to mental health, like health topics and emotion words. This research empirically demonstrates the potential and limitations of current privacy preservation techniques for mental health inference tasks.
翻译:摘要:社交媒体文本数据常被用于训练机器学习模型,以识别表现出高风险心理健康行为的用户。然而,共享此类敏感数据会带来隐私风险,并限制了基准数据集的增长。我们全面评估了隐私保护机器学习技术能否在保持模型性能的同时实现更安全的数据共享。具体而言,我们针对两项广泛研究的心理健康预测任务:X(推特)上的抑郁症检测和Reddit上的自杀危机检测,应用了联邦学习(FL)和差分隐私联邦学习。通过将每个用户视为非独立同分布场景下的客户端,我们模拟了真实的数据共享情景,评估了不同客户端比例、聚合策略和隐私预算下的表现。结果表明,在抑郁症识别中,联邦学习达到了与集中训练(集中训练的F1值为85.63;最佳FL模型的F1值为83.16)相当的性能,但我们发现差分隐私联邦学习即使加入低水平噪声(epsilon=50)也显示出巨大的性能-隐私权衡(F1值下降高达27.01)。这源于与心理健康相关且高度信息丰富但稀疏的语言标记(如健康主题词和情感词)的扭曲。本研究实证展示了当前隐私保护技术在心理健康推断任务中的潜力与局限性。