Well curated, large-scale corpora of social media posts containing broad public opinion offer an alternative data source to complement traditional surveys. While surveys are effective at collecting representative samples and are capable of achieving high accuracy, they can be both expensive to run and lag public opinion by days or weeks. Both of these drawbacks could be overcome with a real-time, high volume data stream and fast analysis pipeline. A central challenge in orchestrating such a data pipeline is devising an effective method for rapidly selecting the best corpus of relevant documents for analysis. Querying with keywords alone often includes irrelevant documents that are not easily disambiguated with bag-of-words natural language processing methods. Here, we explore methods of corpus curation to filter irrelevant tweets using pre-trained transformer-based models, fine-tuned for our binary classification task on hand-labeled tweets. We are able to achieve F1 scores of up to 0.95. The low cost and high performance of fine-tuning such a model suggests that our approach could be of broad benefit as a pre-processing step for social media datasets with uncertain corpus boundaries.

翻译：经过精心策展的大规模社交媒体语料库，包含广泛的公众舆论，可作为补充传统调查的替代数据源。尽管调查在收集代表性样本和实现高准确率方面行之有效，但其成本高昂且响应时间滞后于舆论数天或数周。这两个缺点均可通过实时、高容量的数据流和快速分析管线予以克服。组织此类数据管线的核心挑战在于设计一种高效方法，以快速选择最相关的文档语料库进行分析。仅使用关键词查询往往包含难以通过词袋自然语言处理方法消除歧义的不相关文档。本文探索了语料库策展方法，利用预训练的Transformer模型过滤不相关推文，并通过人工标注的推文针对二分类任务对模型进行微调。我们实现了最高达0.95的F1分数。微调此类模型的低成本和高性能表明，该方法作为预处理步骤，对语料库边界不确定的社交媒体数据集具有广泛的实用价值。