This study presents a multi-stage classification framework for detecting human values in noisy Russian language social media, validated on a random sample of 7.5 million public text posts. Drawing on Schwartz's theory of basic human values, we design a multi-stage pipeline that includes spam and nonpersonal content filtering, targeted selection of value relevant and politically relevant posts, LLM based annotation, and multi-label classification. Particular attention is given to verifying the quality of LLM annotations and model predictions against human experts. We treat human expert annotations not as ground truth but as an interpretative benchmark with its own uncertainty. To account for annotation subjectivity, we aggregate multiple LLM generated judgments into soft labels that reflect varying levels of agreement. These labels are then used to train transformer based models capable of predicting the probability of each of the ten basic values. The best performing model, XLM RoBERTa large, achieves an F1 macro of 0.83 and an F1 of 0.71 on held out test data. By treating value detection as a multi perspective interpretive task, where expert labels, GPT annotations, and model predictions represent coherent but not identical readings of the same texts, we show that the model generally aligns with human judgments but systematically overestimates the Openness to Change value domain. Empirically, the study reveals distinct patterns of value expression and their co-occurrence in Russian social networks, contributing to a broader research agenda on cultural variation, communicative framing, and value based interpretation in digital environments. All models are released publicly.
翻译:本研究提出了一种多阶段分类框架,用于在嘈杂的俄语社交媒体数据中检测人类价值观,并在750万条随机抽取的公开文本帖子样本上进行了验证。基于施瓦茨的人类基本价值观理论,我们设计了一个多阶段流水线,包括垃圾邮件与非个人内容过滤、价值观相关与政治相关帖子的针对性选择、基于大语言模型的标注以及多标签分类。我们特别关注验证大语言模型标注及模型预测结果相对于人类专家的质量。在此过程中,人类专家标注未被视作绝对真相,而被视为一个具有自身不确定性的解释性基准。为应对标注的主观性,我们将大语言模型生成的多个判断聚合为反映不同一致程度的软标签。随后利用这些软标签训练基于Transformer的模型,使其具备预测十种基本价值观各自概率的能力。表现最佳的模型为XLM-RoBERTa-large,在保留测试数据上实现了宏平均F1值0.83和F1值0.71。通过将价值观检测视为一项多视角解释性任务——其中专家标签、GPT标注与模型预测代表了对相同文本具有一致性但非完全相同的解读,本研究表明,模型总体上与人类判断相符,但系统性地高估了“对变化的开放性”这一价值观域。实证层面,该研究揭示了俄罗斯社交网络中价值观表达及其共现的独特模式,为数字环境中文化差异、沟通框架及基于价值观的解读等更广泛的研究议程做出了贡献。所有模型均已公开发布。