In religion and theology studies, spirituality has garnered significant research attention for the reason that it not only transcends culture but offers unique experience to each individual. However, social scientists often rely on limited datasets, which are basically unavailable online. In this study, we collaborated with social scientists to develop a high-quality multimedia multi-modal datasets, \textbf{SACRED}, in which the faithfulness of classification is guaranteed. Using \textbf{SACRED}, we evaluated the performance of 13 popular LLMs as well as traditional rule-based and fine-tuned approaches. The result suggests DeepSeek-V3 model performs well in classifying such abstract concepts (i.e., 79.19\% accuracy in the Quora test set), and the GPT-4o-mini model surpassed the other models in the vision tasks (63.99\% F1 score). Purportedly, this is the first annotated multi-modal dataset from online spirituality communication. Our study also found a new type of connectedness which is valuable for communication science studies.
翻译:在宗教与神学研究中,灵性因既超越文化又为个体提供独特体验而受到广泛学术关注。然而,社会科学家通常依赖有限的、基本无法在线获取的数据集。本研究通过与社会科学家的跨学科合作,开发了高质量多模态数据集SACRED,其分类可靠性得到保障。基于SACRED,我们评估了13种主流大语言模型及传统规则方法与微调方法的性能。结果表明,DeepSeek-V3模型在抽象概念分类任务中表现优异(Quora测试集准确率达79.19%),而GPT-4o-mini模型在视觉任务中以63.99%的F1分数超越其他模型。据称,这是首个源自在线灵性交流的标注多模态数据集。本研究还发现了一种对传播学具有重要价值的全新连接类型。