This paper presents an empirical study to build relation extraction systems in low-resource settings. Based upon recent pre-trained language models, we comprehensively investigate three schemes to evaluate the performance in low-resource settings: (i) different types of prompt-based methods with few-shot labeled data; (ii) diverse balancing methods to address the long-tailed distribution issue; (iii) data augmentation technologies and self-training to generate more labeled in-domain data. We create a benchmark with 8 relation extraction (RE) datasets covering different languages, domains and contexts and perform extensive comparisons over the proposed schemes with combinations. Our experiments illustrate: (i) Though prompt-based tuning is beneficial in low-resource RE, there is still much potential for improvement, especially in extracting relations from cross-sentence contexts with multiple relational triples; (ii) Balancing methods are not always helpful for RE with long-tailed distribution; (iii) Data augmentation complements existing baselines and can bring much performance gain, while self-training may not consistently achieve advancement to low-resource RE. Code and datasets are in https://github.com/zjunlp/LREBench.

翻译：本文针对低资源场景下的关系抽取系统构建进行了实证研究。基于近期预训练语言模型，我们全面探究了三种评估低资源环境下性能的方案：(i) 基于不同类型提示方法结合少样本标注数据；(ii) 针对长尾分布问题的多样化平衡方法；(iii) 通过数据增强技术和自训练生成更多领域内标注数据。我们构建了涵盖8个关系抽取数据集的基准测试，这些数据集覆盖不同语言、领域和上下文，并对所提方案及其组合进行了广泛比较。实验表明：(i) 尽管基于提示的微调在低资源关系抽取中有效，但仍有显著提升空间，尤其在跨语句上下文提取包含多个关系三元组时；(ii) 平衡方法并非始终有益于长尾分布的关系抽取任务；(iii) 数据增强可补充现有基线并带来显著性能提升，而自训练在低资源关系抽取中未必能持续取得进展。代码与数据集详见 https://github.com/zjunlp/LREBench。