The rapid diversification of social media platforms and the increasing restrictions on official APIs have significantly complicated cross-platform analysis. Researchers are often forced to rely on heterogeneous datasets obtained through web scraping and historical archives; however they often lack structural consistency. Prior to conducting cross-platform social media analyses, one needs to answer three critical questions: (1) What makes platforms different and similar? (2) How were the datasets collected? (3) How can we align the datasets of different platforms to conduct fair analyses? To address these questions, we introduce the Social Media Data Toolkit (\projectname{}), a comprehensive Python framework designed for the standardization, anonymization, and enrichment of social network datasets. \projectname{} unifies diverse data structures into a generic schema comprising Communities, Accounts, Posts, Actions, and Entities to facilitate multi-platform research. The framework features a configurable anonymization module to secure Personally Identifiable Information (PII) and an extendable enrichment layer that integrates Large Language Models (LLMs) and network analysis tools for downstream tasks such as stance detection and toxicity scoring without creating codebase for different datasets. We demonstrate the versatility of \projectname{} through four case studies spanning from textual analysis of the content to network analysis across platforms. To offer reproducible social media research, \projectname{} is released as an open-source tool featuring detailed documentation and practical guides for researchers at any skill-level. It can be accessed at github.com/ViralLab/SMDT and varollab.com/SMDT.
翻译:社交媒体平台的快速多样化以及官方API日益严格的限制,使得跨平台分析变得极为复杂。研究人员往往被迫依赖通过网络爬取和历史存档获取的异构数据集,但这些数据集通常缺乏结构一致性。在进行跨平台社交媒体分析之前,需要回答三个关键问题:(1) 各平台的差异与相似之处何在?(2) 数据集是如何采集的?(3) 如何对齐不同平台的数据集以进行公平分析?为解决这些问题,我们介绍了社交媒体数据工具包(\projectname{}),这是一个全面的Python框架,专为社交网络数据集的标准化、匿名化和丰富化设计。\projectname{} 将多样化的数据结构统一为包含社区、账户、帖子、动作和实体在内的通用模式,以促进多平台研究。该框架配备了一个可配置的匿名化模块,用于保护个人身份信息(PII),并提供一个可扩展的丰富层,集成大语言模型(LLMs)和网络分析工具,用于下游任务(如立场检测和毒性评分),无需为不同数据集创建代码库。我们通过四个案例研究展示了\projectname{}的多功能性,涵盖从内容文本分析到跨平台网络分析的各个层面。为实现可复现的社交媒体研究,\projectname{}作为开源工具发布,附带详细的文档和实用指南,适用于任何技能水平的研究者。可通过github.com/ViralLab/SMDT和varollab.com/SMDT访问。